Select Committee on Science and Technology Appendices to the Minutes of Evidence


Memorandum submitted by the Institute of Biology and the British Association of Cancer Research, British Association for Lung Research, British Ecological Society, British Electrophoresis Society, British Society for Plant Pathology, British Society for Soil Science, Laboratory Animal Science Association, Royal Entomological Society, and the Society for Applied Microbiology

  1.  This response's principal points include:

(i)  Generally, the RAE confirms that UK research is extremely competitive internationally in terms of scientific publications.

(ii)  However it is impossible to tell if there has been a genuine improvement in research performance but there has been an increased ability to meet RAE criteria.

(iii)  Secondary "scholarship" criteria do not have sufficient prominence.

(iv)  Industrial, applied and policy driven university research are not recognised and so is disadvantaged by the RAE.

(v)  Quality of research does not necessarily equate with cost of research.

(vi)  Though improved compared with the last RAE there are still concerns over collaborative research and that the RAE encourages short-term research.

(viii)  Those taking maternity career breaks are currently disadvantaged.


RAE indicates UK research excellent value in scientific publications terms

  2.  The RAE provides an affirmation that the UK research is of excellent value in terms of scientific publications and their impacts. As such it provides corroboration of Professor Sir Robert May's (the previous Chief Scientific Advisor's) analysis that the UK research is most cost-effective compared to that of its competitors. UK research provides an excellent return for the investment made in it and so warrants nurturing.


Difficult to tell but it is almost certain that departments' RAE strategy has improved

  3.  It is difficult to tell in a practical sense whether there has been a real improvement in UK research quality even though the RAE scores indicate this. Clearly there have been discoveries and developments, but that it the nature of research. It is impossible to tell whether the rate and/or nature of this progress have improved. However a view common to nearly all who submitted comments to this response was that some or all of this improved RAE score is undoubtedly due to increased familiarity with RAE exercises and the ability of university departments to play the RAE game. This was something that many of the biological Affiliated Societies warned the HEFC in March 1998.

Our 1996 advice to combat gamesmanship has proven to be prudent

  4.  The Institute and a number of the Affiliated Societies suggested in their 1998 response to the HEFCs' pre-RAE consultation that greater standardisation across the Units of Assessment (UOAs) as well as simplification, as a way to combat gamesmanship. It would appear that, given the 2001 RAE results, this was prudent advice as, though there was improvement compared to previous RAEs, these gains in general (but not always) were across subjects and universities. This meant that nearly everyone played a better RAE game such that gamesmanship improvements largely cancelled out. Broadly scores improved equally.

In a strict technical sense it is not possible to compare scores from different RAE years

  5.  It should be noted that technically it is impossible to categorically compare RAEs from previous years as though broadly they are all based on the assessment of publications, grant levels, student numbers, etc each has been different in a number of ways. Consequently only general comparisons can be made.

Journal numbers have increased so paper publication would increase anyway

  6.  Given that journal numbers have increased since 1986, there are more opportunities to publish so there is an argument that any increase in scientific paper publication would have happened anyway.


Yes. Secondary scholarship criteria are not considered with sufficient strength

  7.  Yes. While it is important that research is assessed and that quality of research is the prime criterion, assessment of what are called scholarship activities (attending symposia, participating in learned society activities, assisting with public consultations on scientific matters, etc) should be very much included as secondary criteria. While the inclusion of activities on external (non-university) committees is included the perception is that this is peripheral to determining the RAE score. Furthermore some other scholarship activities are simply not considered. We have heard on a number of occasions of Departments that have actively instructed staff not to engage in any activity that does not directly lead to the improvement of the Department's RAE score.

Scholarship activities are vitally important

  8.  Scholarship activities are vitally important for putting work into context, obtaining synergistic benefit through discussion with other researchers, by furthering a researcher's specialism, and in making expertise available to policy-makers and Government in return for taxpayer support.

Though useful, there still remain concerns as to the domination of citation in the RAE

  9.  There are still the concerns that over-reliance, directly or indirectly, on citation analysis and citation impact factors is not an accurate reflection of research activity or excellence. Whether for example citation indices or journal impact factors are used directly or not, the pressure on individual researchers is to publish in outlets with a "high rating". How is this being judged and by what means? This places an over-emphasis on particular types and forms of output. Only just this month, to give a contemporary example, of how this may be misleading even when done "formally", the journal Nature expressed concern over errors in citation statistics (on which to some extent the RAE depends). Nature was concerned that the paper by the International Human Genome Sequencing Consortium was given such a low citation count that it was absent from the lists of "hot papers" in biology. The Nature editorial (v415, pp101) concludes that "researchers, policy-makers, and publishers who depend heavily on citation statistics should be urged to treat them with greater caution".

Industrial research is not assessed but industry refers to the RAE

  10.  We are still concerned that industrially funded university research is not assessed. The argument against the inclusion of industrially funded research in the RAE is that without publications it is difficult to assess and that the RAE is used to assess Government funded research. However the reality on the ground is that the RAE does have a substantial influence on where industry places research. Furthermore there are options for some form of industrial assessment. (For instance, as suggested in 1998, independent financial auditors could affirm how much industrially funded work a Department had conducted. This at least would provide a cash indicator.) The quantity and quality of such research cannot sensibly be ignored if RAE is to have realistic meaning.

Quality of research does not equal cost of research

  11.  It should also be remembered that the quality of research as indicated by judgement of publication in outlets which are recognised as having a high reputation (high citation rates and impact factors) from which funding follows, does not necessarily reflect the cost of research. For example, in the biosciences work with animals rightly necessitates welfare costs and is not always cheap. Such work is invariably of scientific significance (otherwise Home Office licences would not be granted) but the work—though sometimes of break through importance—may not always generate an Earth-shattering journal paper capable of boosting an RAE score or may only do so over an extended period of time. This is not the type of research that a Head of Department would encourage colleagues to undertake with a view to RAE scrutiny.

RAE determines infrastructure funding but other factors need to be included

  12.  Research infrastructure remains a problem: in the past funding determined by RAEs results as well as from other sources failed to meet infrastructure needs and so infrastructure declined. Currently, the worst of the immediate infrastructure problem has been greatly eased by the Joint Infrastructure Fund as well as the Science Research Infrastructure Fund initiatives. These initiatives are greatly welcome and send an encouraging message to both the UK scientific community as well as multinational industry that can choose with which nations research investment will be placed. However, there remains the longer-term problem. The RAE may well have a part to play in how infrastructure funds are distributed but other factors will undoubtedly have to be included, and of course resources in the longer term need to be made on an on-going basis. This is an area that warrants further consideration.

The RAE encourages short-term research

  13.  There are still concerns that the RAE encourages research capable of generating high-impact papers quickly. Departments are therefore discouraged from under-taking longer-term research despite the possibility that some of such research may be of greater value to the nation, its economy, environmental quality and/or human well-being. Concerns over long-term research were cited in the Affiliated Societies Science Policy Priorities 2001. Similar RAE concerns have been expressed by others including a British Medical Journal editorial (320:636-637) with regards to health sector research. (Here bioscientists work on biomedical research.)

There are still collaboration concerns though matters were better in the last RAE

  14.  Despite reassurances there are still concerns that research arising out of inter-Departmental (especially between universities) collaboration is not properly recognised by the RAE, though steps have been taken to improve matters for the most recent RAE. This concern also covers, of course, highly prized international collaboration.


Is environmental science anomalous?

  15.  A recent survey of the biological Affiliated Societies has questioned whether we currently have the correct balance between whole-organism and molecular research. Both branches of biology are important and are especially illuminating if one is seen in the context of the other. However it may be that there has in recent years been e more focus on one to the detriment of the other. In the bioscience spectrum, the areas furthest away from molecular biology are perhaps the disciplines of ecology and environmental science. Notwithstanding that environmental science is a recent area of research its Departments tend not to have been awarded as high a score as other Units of Assessment (UOAs). It is not clear as to why this might be. It may be: that the quality of such research is genuinely anomalous; or that being both a multi and interdisciplinary research activity that it fell foul of collaborative Departmental RAE concerns; or that the academic interests of the UOA panel members did not reflect that of areas of environmental research; or it might be that much environmental science research is policy-driven rather than the fundamental or blue skies which is at the RAE's focus; or a combination of such factors; or something else. However we believe that this is worth following up.


An independent audit of the RAE's utility is required

  16.  A full and independent audit of the cost and benefit of the RAE to the nation is required. The audit needs to be conducted by those unconnected with the UK university system. Only through such an audit can we determine how the RAE has been successful, and where it has let researchers down. Indeed at a cost to the nation of some £40 million (equivalent to four per cent of the annual Funding Councils' infrastructure grants) the RAE needs to be seen to be providing genuine value for money.

The RAE does not assess research across the board and so needs a name change

  17.  The RAE, if it remains as it stands, needs a name change. It does not assess UK research per se across the board. It only assesses university research, and even here it does not fully account for applied or industrial research (particularly where confidentiality is a highly imposed condition), nor does it properly account for policy-driven (Government Departmental) research. There is a tension between any assessment of "quality" and the impact factor that the research has in reality, in other words and its usefulness and application. This is common in the biosciences where research not only can be blue skies and fundamental (which is the RAE's prime focus) but applied (humans are living creatures dependent on living creatures). For example in plant pathology, to take just one specialist area, basic research is often required to solve real problems in industry. While the results may be incredibly useful, they will not necessarily result in publication in a high impact journal and/not within a modest timeframe. Indeed it is widely accepted that no (or very few) agricultural journals have a high impact though we all depend on food in a World with a growing human population.

Clarity as to what exactly is being assessed is required

  18.  It needs to be made far clearer as to actually what is being assessed by the RAE. We repeat our conviction that the quality of research should remain the prime criterion but in clear conjunction with scholarship activities as the secondary criterion, with the final score being determined by both criteria.

A separate score is required for industrial and policy-driven research

  19.  It may be that an assessment or assessments with separate scores need to be simultaneously made of Departments' industrial and policy-driven research. The Funding Councils may not welcome having to do this but it would be far more efficient to have one all-encompassing assessment delivering two or three scores that revealed Departments' various strengths in different types of work.

Take time to devise a fresh assessment mechanism

  20.  The RAE has provided some benefit but has arguably had its day in its existing form. It is important that the next Research Assessment is a clear development of previous RAEs. Care and time should be taken to formulate it. Clearly given the general improvement across the board in RAE score, there would be little to be gained from repeating exactly the same exercise in five years time. Therefore there would be little harm, and much benefit, in waiting for a completely re-vamped assessment in 7 to 10 years time. (This would also enable researchers to focus in the interim on research and free them of RAE constraints.)


Problems with being declared research inactive following a maternity break

  21.  There is general concern as to the state of women in science. Of the three pure sciences, biology has the greatest proportion of women in its younger cohorts. However in common with all of science there is the problem that if someone takes a career break to start a family and then returns between RAEs, then they may be declared research inactive. This issue is surely resolvable and needs to be addressed.

Some subjects may fall between Units of Assessment

  22.  There are concerns that some subject areas such as microbiology and environmental science are falling between Units of Assessment. Their subject areas make them capable of being assessed by more than one UOA and end up not being properly assessed by either. It is important that areas of research are not turned away from being properly assessed.

Consultation deadline concerns

  23.  There was general dismay at the way this consultation was only afforded a month (one third of the Cabinet Office minimum guidelines) and was launched just prior to the Christmas/New Year break (itself contrary to the guidelines). This does not facilitate confidence in open and fair consultation and made it extremely difficult to obtain and approve views from key constituent representatives of the UK biological community.

22 January 2002

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2002
Prepared 24 April 2002