Peer review in scientific publications - Science and Technology Committee Contents


Written evidence submitted by Professor Howard Elcock, AcSS (PR 06)

SOME PROBLEMS WITH THE ANONYMOUS REFEREEING SYSTEM IN THE UK SOCIAL SCIENCES

1.  I am Professor (emeritus) of Government at Northumbria University As an academic specialising in Political Science and Public Administration of some 40 years' standing, I have had extensive experience both of submitting articles and research projects for refereeing. I have also acted as such a referee myself on many occasions, especially as a member of the Editorial Board of the Journal Public Policy & Administration.

2.  Some years ago the Royal Society expressed concern about the working of the anonymous refereeing system which is now in general use for assessing manuscripts submitted for publication in learned journals, as well as to vet applications for research funding. The Royal Society appointed a working party chaired by Lord Rees of Ludlow, to investigate this and other problems concerned with disseminating scientific knowledge to the general public. In the light of the Society's concern about the refereeing system, I raised this issue with the Committee of Academicians of the Academy of Social Sciences. The ensuing discussion made it clear that my occasionally adverse experiences were by no means unique. The Royal Society's working party (Royal Society, 2006) confirmed the existence at least some of the problems indicated in this submission. The present document is a revised and updated version of that submission.

3.  The first major problem is that unlike the United States and possibly other countries, British communities of scholars are usually too small for refereeing to be truly anonymous: one Editor has said to me that "the small world we inhabit certainly presents real difficulties in securing genuinely anonymous refereeing, especially in some areas of specialised work". For example the Political Studies Association, which is probably among the larger British social science professional bodies, had in 2006 some 1,600 members, which constitutes a considerable majority of political scientists teaching in British universities and other higher education institutions (see PSA, 2006). While this is perhaps too many colleagues for any of us to be personally acquainted with all of them and their work, this would not apply to the more senior members who are likely to be both the more extensively published researchers and to be asked most often to act as anonymous referees - they are perhaps between a third and a quarter of the total. They may well personally know most of those involved in their research fields and be able to identify their work. Furthermore, when the general academic community in a particular discipline is broken down into its specialist research fields, it would be quite possible for all the researchers in a particular field to know, or at least to be aware of one another's work. For instance, research on governance and strategic planning in the English regions was being conducted in the early years of the last decade by only about a dozen researchers or research teams, two of which were nationally and internationally eminent. Thus researchers in this specialist field could probably identify one another's work when asked to referee it anonymously. Hence professional rivalries or personal animosities might distort a referee's judgment This danger has been made more serious by the attempts by the Thatcher Government and its successors to inject commercial competition into academic life.

4.  Secondly, referees have been known simply to dismiss a colleague's work as rubbish without explaining themselves or giving any reasons for dismissive judgments. Not only does this mean that the author, whoever he or she may be, can derive no constructive benefit from their comments; it is also extremely distressing for them. It might cause a young researcher to give up academic work or even to resort to self-harm. Such brief dismissals of someone's work constitute a dereliction of the referee's scholarly duty to assist colleagues in developing their research.

5.  There is a possible danger that the refereeing system may be used to enforce academic orthodoxies and deny dissident or challenging researchers their right to publish. This may be related to the problem of researchers being denied publication because their findings are unwelcome to their research sponsors or their Head of Department (CAFAS, passim). The Universities and Colleges Union has to deal with such cases form time to time.. In this context, referees who criticise an author for not properly reviewing "the literature" should be regarded with some suspicion. An editor has said that for his journal, refereeing is intended mainly to ensure the "validity and comprehensibility" of submitted articles: perhaps such a limiting ordinance should be made more general. One effect of refereeing has been that public academic controversies of old, such as that between AJP Taylor and High Trevor Roper over the origins of the Second World War, have largely vanished, which may (or may not) be a cause for regret.

6.  A major fault with the refereeing system is the long delays that seem to be endemic in it, a point also made by the Royal Society, whose committee stated that refereeing "has been subject to criticism from outside the research community for delaying or even preventing the disclosure of research results that may have a bearing on the public interest" (2006: 12). Senior academics, who bear the brunt of refereeing, are busy people but they ought to be able and willing to turn requests for refereeing journal articles around within around two weeks of receiving them. One hopes that these delays never have an ulterior motive but they are too common for delay not to be regarded as a significant weakness in the system.

7.  Referees can make mistakes and of course, their judgments must inevitably be subjective. One referee may be severely critical of a manuscript, while another might recommend its acceptance for publication cum laude.

8.  Where a journal is seeking contributions from practitioners as well as academics, as is the case, for example, with the JUC/PAC journal, Public Policy & Administration, submitting their contributions to referees may not be acceptable to senior civil servants or local government officers who are asked or encouraged to prepare contributions. This may also be the case with Public Money & Management, which has a large practitioner audience and seeks contributions from practitioners of public administration and management, as well as from academics in the field. Also, the delays involved in refereeing may mean that such contributions will lose their topicality as a result of the inevitable delay in publishing their work. In such cases, a flexible editorial policy may be needed but not at the risk of endangering the journal's credibility.

9.  Possible components of a solution might include:

—  Permitting or requiring referees to sign their reports, which would also enable referees, if they saw fit, to pass the manuscript and their comments on them back to authors in the hope that they might be helpful. The Committee and learned societies might go a bit further than the Royal Society originally proposed. They suggested a voluntary system whereby referees would be able to sign their reports if they wished (2006: 12), in the interest of transparency, although some referees may not wish to do so and some might not be prepared to act as referees if they are asked to sign their reports. It would not, therefore, be practicable to make signed reports obligatory but it could be strongly encouraged.

—  Editors of journals and research funders could follow the Economic and Social Research Council's example and ask authors themselves to nominate one or two possible referees. The editors or research funders could then nominate additional or alternative referees as they thought fit but at least authors would then have some reassurance that they might have "a friend at court". However, all referees must be instructed to deliver honest, frank assessments of authors' work.

—  Editors must be much firmer about setting and enforcing deadlines for the receipt of referees' reports. I see encouraging signs that some editors are trying to do this. If a referee has not completed his or her report within, say, a month, the editor should withdraw the request for refereeing and ask someone else to undertake it. In extreme cases, one referee's report plus the editor's own judgment may have to be sufficient to determine a MS's fate.

REFERENCES

CAFAS, passim: Campaign for Academic Freedom and Standards, CAFAS Update, published periodically, contains numerous accounts of the suppression of research by Heads of Department, other university authorities or research sponsors (usually Government departments).

Political Studies Association, 2006: Members' Handbook. 2006.

Royal Society, 2006: Science and the public interest: communicating the results of new scientific research to the public.

Professor Howard Elcock

21 February 2011


 
previous page contents next page


© Parliamentary copyright 2011
Prepared 28 July 2011