Conclusions and recommendations
The peer-review process
1. We
conclude that different types of peer review are suitable to different
disciplines and research communities. We consider that publishers
should ensure that the communities they serve are satisfied with
their choice of peer-review methodology. Publishers should keep
them updated on new developments and help them experiment with
different systems they feel may be beneficial. (Paragraph 20)
2. The importance
of a pre-publication technical assessment is clear to us. It should
be a fundamental aim of the peer-review process that all publications
are scientifically sound. Assessing the impact or perceived importance
of research before it is published will always require subjective
judgement and mistakes will inevitably be made. We welcome new
approaches that focus on carrying out a technical assessment prior
to publication and making an assessment of impact after publication.
(Paragraph 29)
3. We recommend that
publishers, research funders and the users of research outputs
(such as industry and Government) work together to identify how
best to evaluate current peer-review practices so that they can
be optimised and innovations introduced, and the impact of the
common criticisms of peer review minimised. We consider that this
would also help address any differences in the quality of peer
review that exist. We encourage increased recognition that peer-review
quality is independent of journal business model, for example,
there is a "misconception that open access somehow does not
use peer review". (Paragraph 58)
Innovative approaches to peer review
4. We
conclude that pre-print servers can be an effective way of allowing
researchers to share and get early feedback on preliminary research.
The system is well established in the physics community, and works
particularly well, co-existing with more traditional publication
in journals. We encourage exploration in other fields. We note,
however, that pre-print servers may not work in fields where commercialisation
and patentability are issues, or in the biomedical sciences, where
publication of badly performed studies could have harmful consequences
and could be open to misinterpretation. (Paragraph 72)
5. The principles
of openness and transparency in open peer review are attractive,
and it is clear that there is an increasing range of possibilities.
There are mixed results in terms of acceptance amongst researchers
and publishers, although some researchers are keen to see greater
transparency in their fields. We encourage publishers to experiment
with the various models of open peer review and transparency and
actively engage researchers in taking part. (Paragraph 78)
6. We are impressed
by the success of PLoS ONE and welcome the wider growth
of quality online repository journals. These will accelerate the
pace of research communication and ensure that all work that is
scientifically sound is published, regardless of its perceived
importance. However, we recognise that this is a relatively new
and rapidly evolving model, and potentially open to abuse because
publication fees are involved. It is important that a high quality
of peer review is maintained across all repository-style journals.
(Paragraph 89)
Editors, authors and reviewers
7. The
role of the editor is at the heart of the peer-review process.
The judgement applied by the editor to the information collected
in the review process requires knowledge, skill, and care; particularly,
in respect of identifying the right reviewers for the job and
critically assessing the feedback from reviewers to authors. (Paragraph
100)
8. Broadly speaking,
training for editors and members of editorial boards is provided
on the job. We have heard that some publishers opt for a more
structured approach, and include, for example, comprehensive welcome
packs for new editors that cover peer-review processes, support
tools and ethical guidelines. We encourage publishers to work
together to develop standards—which could be applied across
the industry—to ensure that all editors, whether staff or
academic, are fully equipped for the central role that they play
in peer review. (Paragraph 106)
9. A relatively straightforward
way of educating reviewers about the quality of their reports
and helping them improve their feedback to editors is to send
them the reports of other reviewers, done confidentially when
necessary. This should be standard practice across all journals.
This would be a useful educational tool to improve the quality
of future reports from reviewers. (Paragraph 118)
10. Training for the
next generation of authors and reviewers is also important. Many
PhD students and post-doctoral researchers are fortunate to have
the opportunity to discuss scientific literature in journal clubs
and other informal settings. Some are mentored well by their principal
investigator and thereby receive informal training in peer review.
Others are not. Given the importance of peer review across the
research spectrum, from grant applications to publications, we
consider that all early-career researchers should be given the
option for training in peer review. (Paragraph 119)
11. Training for early-career
researchers is important. We note that "Roberts Funding"
is coming to an end and that the Research Councils will therefore
be increasing the amount they give to universities "for training
and developing postgraduate research". We invite the Research
Councils to set out further details of how and where this money
will be allocated and what proportion of it will be dedicated
to training in peer review, including academic writing and publication
ethics (discussed later in this report). We also ask for further
details of how this will be "joined up" across different
research funders. (Paragraph 124)
12. We welcome the
fact that the publishers we have heard from are training authors
and reviewers on an international level, particularly those from
countries which are not traditional scientific leaders, and we
encourage others to do the same. This should help alleviate the
current imbalance between publication output and participation
in peer review. (Paragraph 130)
The burden of reviewing
13. We
are not convinced that there is a "crisis" in the supply
of reviewers, especially as so little data are available. It appears
that the current imbalance between publication output and participation
in peer review may be a transitory phase. However, publishers
should not be complacent and should continue actively to monitor
the situation by collecting data. (Paragraph 134)
14. Peer review is
a burden on researchers but a necessary one, as it is an integral
part of the scientific and research process and is part of the
role of a researcher. However, we encourage publishers to work
with their reviewers, to identify innovative new practices to
minimise the burden. (Paragraph 152)
15. In order to help
research institutions recognise the work carried out by reviewers
on peer review, publishers first need to have in place systems
for recording and acknowledging it. A variety of approaches are
in use, including rewards, awards and letters of endorsement and
these should be encouraged. New initiatives for accurate author
and reviewer identification may make it easier for publishers
to track reviewer contribution to the peer-review process. (Paragraph
164)
The assessment of researchers and institutions
16. We
have concerns about the use of journal Impact Factor as a proxy
measure for the quality of an individual article. We have been
reassured by the research funders that they do not consider that
publication in a high-impact journal should be used as a proxy
measure for assessing either the work of individual researchers
or research institutions. We agree that there is no substitute
for reading the article itself in assessing the worth of a piece
of research. We consider that there is an element of chance involved
in whether researchers are able to get their articles published
in high-impact journals, depending on topicality and other factors.
Research institutions should be cautious not to attach too much
weight to publication in high-impact journals when assessing individuals
for career progression. (Paragraph 177)
Managing data
17. We
conclude that reproducibility should be the gold standard that
all peer reviewers and editors aim for when assessing whether
a manuscript has supplied sufficient information, about the underlying
data and other materials, to allow others to repeat and build
on the experiments. (Paragraph 184)
18. If reviewers and
editors are to assess whether authors of manuscripts are providing
sufficient accompanying data, it is essential that they are given
confidential access to relevant data associated with the work
during the peer-review process. This can be problematical in the
case of the large and complex datasets which are becoming increasingly
common. The Dryad project is an initiative seeking to address
this. If it proves successful, funding should be sought to expand
it to other disciplines. Alternatively, we recommend that funders
of research and publishers work together to develop similar repositories
for other disciplines. (Paragraph 189)
19. Access to data
is fundamental if researchers are to reproduce, verify and build
on results that are reported in the literature. We welcome the
Government's recognition of the importance of openness and transparency.
The presumption must be that, unless there is a strong reason
otherwise, data should be fully disclosed and made publicly available.
In line with this principle, where possible, data associated with
all publicly funded research should be made widely and freely
available. Funders of research must coordinate with publishers
to ensure that researchers disclose their data in a timely manner.
The work of researchers who expend time and effort adding value
to their data, to make it usable by others, should be acknowledged
as a valuable part of their role. Research funders and publishers
should explore how researchers could be encouraged to add this
value. (Paragraph 203)
Post-publication review and commentary
20. Post-publication
review in an era of new media and social networking tools, such
as Twitter, is very powerful. The widespread sharing of links
to articles ensures that research, both accurate and potentially
misleading, is rapidly spread across the world. Failings in peer
review can, rightly, be quickly exposed. However, there is no
guarantee that false accusations of failings will not also be
spread. Pre-publication peer review still has an important role
to play, particularly in relation to assessing whether manuscripts
are technically sound prior to publication. However, we encourage
the prudent use of online tools for post-publication review and
commentary as a means of supplementing pre-publication review.
(Paragraph 211)
21. While it is too
early to make a judgement on post-publication filtering mechanisms,
such as Faculty of 1000 Ltd, we recognise that such a system could
offer a valuable service if widely used. It is likely that such
services will become more important with the growth of repository-type
journals. (Paragraph 223)
Publication ethics and research integrity
22. The
integrity of the peer-review process can only ever be as robust
as the integrity of the people involved. Ethical misconduct damages
peer review and science as a whole. Although peer review is not
designed to identify systematically fraud or misconduct, it does,
on occasion, identify suspicious cases. Where ethical misconduct
is suspected, guidance for journal editors is in place, for example
from the Committee on Publication Ethics, about how best to deal
with it. In addition to relying on the vigilance of the people
involved in the process, publishers must continue to invest in
new technology that helps to identify wrongdoings. (Paragraph
244)
23. Oversight of research
integrity in the UK is in need of revision. The current situation
is unsatisfactory. We are concerned that the UK does not seem
to have an oversight body for research integrity that provides
"advice and support to research employers and assurance to
research funders", across all disciplines. The UK Research
Integrity Futures Working Group report made sensible recommendations
about the way forward for research integrity in the UK. Research
Councils UK, Universities UK and the Government should revisit
these recommendations and reassess how they can best be implemented.
(Paragraph 262)
24. Employers must
take responsibility for the integrity of their employees' research.
However, we question who would oversee the employer and make sure
that they are doing the right thing. In the same way that there
is an external regulator overseeing health and safety, we consider
that there should be an external regulator overseeing research
integrity. We recommend that the Government set out proposals
on the scope and powers of such a regulator and consult with the
research community and other relevant parties to develop them.
(Paragraph 271)
25. We also recommend
that all UK research institutions have a specific member of staff
leading on research integrity. Such a person would be a first
point of call in case of an ethical breach. Where an investigation
subsequently takes place within a research institution, it is
essential that the outcome be published. (Paragraph 272)
26. We recommend that
the Research Councils, and other funders of research, reassess
the robustness of their procedures for dealing with allegations
of research fraud or misconduct, to ensure that they are not falling
through the cracks. (Paragraph 276)
General conclusions
27. Peer
review in scholarly publishing, in one form or another, is crucial
to the reputation and reliability of scientific research. Pre-publication
peer review has evolved in a piecemeal and haphazard way to meet
the needs of individual scientific communities. The process, as
used by most traditional journals prior to publication, is not
perfect, and it is clear that considerable differences in quality
exist. However, despite the many criticisms and the little solid
evidence on its efficacy, editorial peer review is considered
by many as important and not something that can be dispensed with.
(Paragraph 277)
28. In order for current
peer-review practices to be optimised and innovative approaches
introduced, publishers, research funders and the users of research
outputs (such as industry and government) must work together.
There is much that can be done to improve the quality of pre-publication
peer review across the board and to better equip the key players
to carry out their roles. We note that new innovations in pre-publication
review are being introduced that have the potential to accelerate
the pace of research communication and avoid duplication of effort
by the research community, with the consequent drain on resources.
Publishers can learn much from one another and should share best
practice where possible—particularly in relation to the
ways in which data are managed and in terms of promoting publication
ethics and research integrity. It is clear that breaches in the
latter damage both the scientific record and public confidence
in science. (Paragraph 278)
29. The publication
of peer-reviewed articles is not only important in terms of maintaining
a robust scientific record, it also has an impact on the careers
of researchers and the reputations of research institutions. We
have been assured by research funders that they do not use journal
Impact Factor as a proxy measure for the quality of research or
of individual articles. However, representatives of research institutions
have suggested that publication in a high-impact journal is still
an important consideration when assessing individuals for career
progression. We consider that research institutions should be
cautious about this approach, because as we have previously noted,
there is no substitute for reading the article itself in assessing
the worth of a piece of research. (Paragraph 279)
30. While pre-publication
peer review continues to play an important role, the growth of
post-publication peer review and commentary represents an enormous
opportunity for experimentation with new media and social networking
tools. Online communications allow the widespread sharing of links
to articles, ensuring that interesting research is spread across
the world, facilitating rapid commentary and review by the global
audience. They also have a valuable role to play in alerting the
community to deficiencies and problems with published work. We
encourage the prudent use of online tools for post-publication
review and commentary as a means of supplementing pre-publication
review. (Paragraph 280)
| 
