Select Committee on Science and Technology Tenth Report

9 Integrity of the publishing process

Peer review

204. Many of the arguments in the debate on scientific publications focus on the issue of peer review: do new developments in the publishing market put it at risk? As is outlined in paragraphs 169—174, we have concluded that they must not. A factor in this debate is the scientific community's capacity for self-policing. All of the academics that we spoke to were confident that they could determine the quality of a research article for themselves. This stands to reason given the fact that it is the same academics who carry out the function of peer review. Ironically it is this facility for self-regulation that calls peer review into question. If academics can distinguish a good article from a bad one by themselves, why do they need another academic to carry out this function for them? From this argument stems the view that peer review is unnecessarily censorious.

205. There are at least three strong arguments, however, for keeping the system of peer review intact. Firstly, volume. As has already been outlined, academics are producing more research articles than ever before: output increases by approximately 3% per year. Whilst academics might have the acumen to determine which of these articles are worth reading, they probably do not have the time to search through the entire output in order to achieve this. The peer review services provided by publishers act as a filter, saving academics time and thus also saving public money. Secondly, peer review gives successful articles a mark of distinction that helps to provide a measure of the academic's and their department's level of achievement. As Procurement for Libraries notes, for the academic, "scholarly publishing in academic journals is essentially about validation of results through the editorial and peer-review process".[351] We heard that the main motivations for academics to publish were career, funding and reputation-based. These incentives to publish would be significantly reduced were the mark of achievement conferred by passing successfully through the peer review process to be abandoned. Thirdly, peer review gives the lay reader an indication of the extent to which they can trust each article (see paragraph 132).

206. The usefulness of peer review to the scientific process is not a guarantee of its quality. We wrote to the Editors of four high-profile journals, Cell, The Lancet, Science and Nature, to ascertain what measures they used to ensure the integrity of the peer review process. Collectively the Editors cited the following measures:

  • Authors are given the opportunity to exclude from consideration any reviewers who are affected by a potential conflict of interest;
  • Reviewers are given the opportunity to disqualify themselves on the basis of a conflict of interest;
  • Articles are sent to a number of reviewers, for example, Cell uses three reviewers per article and The Lancet uses four. This allows for the moderation of their findings;
  • Editors track all the reviews submitted by a particular reviewer for consistency. Any comments that are judged to be unduly harsh or lenient within that context are noted;
  • Editors evaluate all claims of reviewer bias or misconduct and appropriate action is taken; and
  • Journals have a formal appeals procedure available for all rejected articles.[352]

207. In addition, peer reviewers have no responsibility for making the final decision about which articles are published, and most of them are unpaid, ensuring that they retain a degree of detachment from the publishing process. All of the above measures attempt to minimise the risk of a compromise to the peer review system. However, as Richard Horton, Editor-in-Chief of The Lancet, pointed out in his response, "these processes rely on the integrity of the individuals involved, and we rely on trust between editors, reviewers, and authors".[353] As is the case with any process, peer review is not an infallible system and to a large extent depends on the integrity and competence of the people involved and the degree of editorial oversight and quality assurance of the peer review process itself. Nonetheless we are satisfied that publishers are taking reasonable measures to main high standards of peer review. Peer review is an issue of considerable importance and complexity and the Committee plans to pursue it in more detail in a future inquiry.

The Research Assessment Exercise

208. The Research Assessment Exercise (RAE) is used as a means of implementing a policy of selective funding for universities. It aims to measure the quality of research in different departments, rewarding excellence where it occurs and encouraging its development elsewhere. The rating awarded to a department by the RAE helps to determine levels of funding. As one of the most readily identifiable and quantifiable research outputs, journal articles are a key measure used by the RAE. What follows is a brief analysis of the impact of the RAE on STM publishing trends. We will examine wider issues concerning the RAE in a forthcoming Report.[354]

209. Publication enhances career and reputation in a general sense: academics do not publish their research findings simply because of the RAE. As Rama Thirunamachandran pointed out in oral evidence, "if you look at other countries which do not have an RAE, people still want to publish in Nature".[355] Nonetheless, we received evidence to suggest that the measures used in the RAE distorted authors' choice of where to publish. Although RAE panels are supposed to assess the quality of the content of each journal article submitted for assessment, we reported in 2002 that "there is still the suspicion that place of publication was given greater weight than the papers' content".[356] This is certainly how the RAE was perceived to operate by the panel of academics we saw on 21 April. Professor Williams told us that he chose to publish in journals with high impact factors because "that is how I am measured every three years or every five years; RAE or a review, it is the quality of the journals on that list".[357] Similarly Professor Crabbe stated that "the driver is finance. The driver is the Research Assessment Exercise. Impact factors, the half-life of journals are what drives us, I am afraid".[358] In both oral and written evidence, HEFCE denied that journal impact factors formed the basis for an assessment of the quality of articles submitted to the RAE.

210. Whether or not RAE panels use journal impact factors as an indication of the quality of the articles that they assess, the perception that this is the case causes a bias amongst UK authors towards journals with higher impact factors. This in turn increases the journal's impact factor still further. In this way, regrettably, the RAE indirectly supports a hierarchy of journals, making it difficult for new and little-known journals, including — because they have appeared only recently — some author-pays journals, to compete. The Open University told us that "Government should encourage the RAE to develop new quality indicators so that articles published in new open access journals can be evaluated in an even-handed manner in the Research Assessment Exercise".[359] However, the current system, which does not formally take account of impact factors, should already ensure that this is the case. The perception that the RAE rewards publication in journals with high impact factors is affecting decisions made by authors about where to publish. We urge HEFCE to remind RAE panels that they are obliged to assess the quality of the content of individual articles, not the reputation of the journal in which they are published.

351   Ev 153 Back

352   Ev 427-8 Back

353   Ev 430 Back

354   HC 586. See also the Second Report of the Science and Technology Committee, Session 2001-02, The Research Assessment Exercise (HC 507) Back

355   Q 397 Back

356   Second Report of the Science and Technology Committee, Session 2001-02, p 17 Back

357   Q 285 Back

358   Q 286 Back

359   Ev 323 Back

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2004
Prepared 20 July 2004