House of Commons - Science and Technology

Select Committee on Science and Technology Eleventh Report

3 Proposals for RAE 2008

13. A key element of Sir Gareth Roberts's review was that it was time to "move away from a 'one-size-fits-all' assessment, to a model which concentrates assessment effort where the stakes are highest".[7] The purpose of different assessment routes (or "tracks") was to impose a bureaucratic burden in proportion to the potential rewards. He proposed three routes:

a) An option of a separate approach for the least research intensive institutions;

b) Assessment by proxy measures against a threshold standard for the less competitive departments in the remainder of institutions; and

c) Expert review assessment similar to the old RAE for the most competitive departments.

14. According to the Funding Bodies, Sir Gareth's model was not endorsed by the consultation responses. The Funding Bodies have accordingly retained a single assessment route. Sir Gareth told us that the responses had been evenly split and that HEFCE were looking for a large majority in favour of change but did not get it.[8] The AUT supports the rejection of Sir Gareth Roberts's tiered system of assessment.[9] The Biosciences Federation says that there is "concern that in multi-track models it may prove difficult to move from one track to a higher one because of funding differentials….[but] under the alternative models institutions with few expectations of high ratings could opt for a less bureaucratic assessment, and that this would relieve pressure on RAE panels".[10] Support for the principle did come from the British Society of Criminology.[11] Sir Howard Newby said he had sympathy with Sir Gareth's assessment routes but he reported "a very strong outcry, from even those institutions which were not strong in research, that they had, if you like, almost an entitlement to be entered for the full RAE, and this was really to do with status rather than money".[12] This was echoed by Professor Richard Joyner: "the idea that the university has to be entered into the RAE to have a spectrum in research at all has acquired a certain macho thing about it, and no vice chancellor is willing to say that they were prepared to accept the other two tracks that Gareth proposed".[13] We regard this as a poor reason for rejecting the multiple track approach.

15. Natalie Fenton's criticism of the multi-track approach was that it "means that you will get a graded system where you will hugely increase the funding differentials that exist already for research".[14] This anticipates the levels of funding for each track. She may have legitimate concerns here given the highly selective funding policies of HEFCE but this should not be seen as a fundamental problem with the three track approach. Steve Wharton told us that the proposed system would have re-established the pre—post-92 divide.[15] To some extent this might have been the case, although some might argue that the three track approach might encourage some of the post-92 universities to pursue distinctive and valuable lines of research, such as those that support local industry, rather than aspire to emulate the basic research activities of the older universities. It should also be appreciated that many post-92 universities boast 5 and 5* departments while many in the older universities continue to underperform. Dr Wharton and Ms Fenton argue that there should be a level of start-up research funding for all departments.[16] It is arguable that the best way of achieving a reasonable level of funding to all departments would be to embrace a multiple track approach.

16. The Funding Bodies should have looked at the quality of the arguments set out in the responses to their consultation and not just the numbers. The move away from the "one size fits all" approach advocated by Sir Gareth Roberts is an important principle which should have been adopted. We consider the Funding Bodies to be unjustifiably conservative in their proposals. We do not see it as HEFCE's role to protect the sensitivities of universities.

17. We also recommended a three track approach in our earlier Report:[17]

i. Top-rated departments would be exempted from the formal research assessment process if they wish. Instead their Funding Council income would be based on their project funding from Research Councils, charities and other sources. Funding levels would need to reflect the source of funding and the overheads included in that funding. HEFCE might reward the individuals responsible for their department's high rating.

ii. Other departments could continue to take part in a research assessment process. Funding from the Funding Councils would then be based on a formula relating research quality and volume as at present but departments not reaching a minimum standard of quality would not be funded.

iii. Departments taking part in the research assessment process could apply for development money through a bidding process and would be assessed by subject panels based upon the RAE units of assessment (UoAs). They would be required to enter subsequent RAEs to provide a benchmark for improvement. Applications would be based on a business plan which should indicate how they intend to achieve a higher research quality rating.

18. This would be combined with a funding stream to encourage research collaboration with external partners. Our system differed in an important respect from Sir Gareth's in that in our system it would be the top-ranked departments that could opt out of the RAE. This is based on the fact that external research income could be used as a more reliable proxy for research quality. It also follows a principle employed by the Government in other areas of public services in which the bureaucratic burden is minimised for the best performing institutions. Our scheme could easily be applied at an institutional level. Not all disciplines are able to attract the same levels of external research income and this would need to be factored in for any calculation of QR funding.

Panels

Structure and functions

19. For RAE 2001, research was divided into 68 subject areas or Units of Assessment (UoAs), of which 32 could be described as science, medicine or engineering. An assessment panel was recruited to examine research in each of these areas. Panel sizes varied according to discipline; for example, Physics had 11 members and Biological Sciences had 20. Eleven subpanels were set up, all in the clinical subjects. Panels could call in outside expertise if its members felt unqualified. A submitting department could ask for its work to be cross-referred to other relevant panels if it spanned the boundary between UoAs or was interdisciplinary in character.

20. Each submission contained the names of "research active staff" along with up to four research outputs for each person; for example, journal articles, books, book chapters, conference contributions and patents. Panels were expected to make a judgement on a researcher based only on the outputs submitted. They were also able to consider reasons why a researcher had not produced the requisite four outputs.

21. In our earlier Report we expressed a number of reservations about how panel members and chairmen are chosen, and about whether, as academics judging other academics, they are truly objective. We were concerned by the size of the panels and the number of outputs they had to consider. These issues have not been addressed by the Funding Bodies in their proposals.

22. In their Initial Decisions, the Funding Bodies announced that the number of main panels would be cut to 15-20 main panels, whose decisions would be based upon detailed assessment work by around 70 sub-panels. In July 2004, the Funding Bodies announced details of the Units of Assessment.[18] There will be 67 subpanels and 15 main panels.

23. Each sub-panel will cover a discrete subject area or unit of assessment, and will report to a single main panel. The role of a main panel will be:

a) To give leadership and guidance to a group of sub-panels on their approach to the assessment process, including approving their criteria for assessment and working methods;

b) To work with the sub-panels during the assessment period to ensure consistent application across the exercise of the overall quality standards, common assessment procedures, and equal opportunities guidance;

c) To sign off quality profiles for all submissions to the sub-panels, based upon the work and advice of the sub-panels;

d) To give advice as requested by the RAE team and funding bodies on aspects of the assessment process, including requests for additional specialist advice; and

e) To produce a final report on the state of research in the disciplines covered by the sub-panels.

24. Each main panel will cover a group of sub-panels. Sub-panels will:

a) Produce draft criteria for assessment and working methods for approval by the main panel;

b) Work within the agreed criteria and in collaboration with the main panel, to produce draft quality profiles (to be signed off by the main panel) and associated brief feedback for all submissions made to them; and

c) Advise the main panel and RAE team on cross-referrals to other sub-panels of submissions or cited research, and on the need for additional specialist advice.

25. These proposals have received a mixed reception. Research Councils UK "support the principle of setting up a system of panels and sub-panels" although Universities UK has "concerns about the complexity of the proposed system and the potential for it to be time-consuming and burdensome" and the Association of the British Pharmaceutical Industry sees the two tier structure as creating even greater bureaucracy.[19] The new system has also been cautiously welcomed by the Biosciences Federation: "The main panel/sub-panel structure may help overcome the previous perceived lack of parity between panel judgements, but more information on panel working is required to predict this with more certainty." However, they go on to say that the "challenging issue of how to deal with cross-disciplinary research is not addressed directly" and "without further detail it is difficult to judge whether cross-disciplinary work, applied research, and joint submissions will be dealt with more satisfactorily than before".[20] The UK Computing Research Committee says "Whether the new RAE structure will be an improvement on the old depends on the panel structure, the criteria that panels will employ, and the relationship between the RAE profile gained and the funding that is awarded" and that it is "essential that panels and sub-panels are free to choose the assessment criteria that are most appropriate for each discipline".[21]

26. In an attempt to promote consistency between panels Sir Gareth proposed that each panel should have a moderator who would sit on each sub-panel with a brief to ensure consistency of practice. The moderators of four or five adjacent panels would sit on a "super-panel" chaired by a senior moderator, who would be seconded to or employed by the funding councils. The Research Councils had "strongly supported the suggestion of setting up each panel with a moderator to ensure consistency of practice across the sub-panels" and was "disappointed that the proposed moderating mechanism no longer figures in the RAE2008 document".[22] Sir Gareth's suggestion appears to have much merit. Sir Howard Newby had suggested that this proposal had not been rejected yet there is no mention of it in the Funding Bodies' detailed proposals on panels published in July 2004.[23] We believe that the panel/sub-panel structure is a positive step in improving consistency and improving the treatment of interdisciplinary research. We recommend that the Funding Councils seriously consider the establishment of panel moderators.

Membership

27. In the past panels were appointed by inviting nominations from professional, trade and learned bodies. In an attempt to broaden the assessment for different forms of research, there has been an attempt in recent RAEs to include panel members from outside the academic community. In our earlier Report we expressed concern about the willingness of representatives from the private sector to serve on panels.

28. Membership of the sub-panels will include people with experience in commissioning and using research—in industry, commerce and the public sector—and people with experience of research in other countries will also attend panel meetings at some stage of the assessment. The Royal Society welcomes the Funding Bodies' commitment to secure better representation on panels of people with personal experience both of conducting research and of its commercial, industrial and public service applications.[24] It is interesting to note that the representation of research users on panels for RAE 2001, as judged by panel chairs, had "little effect on the final grades but were useful for credibility".[25] If assessors from the user communities are aware of this view then it could undermine the Funding Bodies' efforts to secure better representation. It would be a matter of concern if this were the case.

International benchmarking

29. In RAE 2001, the Funding Councils introduced an international benchmarking exercise in which the ratings of all 5 and 5* departments were validated by 290 overseas experts. All but nine agreed with the judgements of the panels. The Funding Bodies expressed few concerns about the use of overseas panel members at the time and we concluded that their use did provide support for the reliability of the grades. In the view of Sir Gareth Roberts's review team, "This improvement [in grades] was validated by the opinion of overseas experts".[26]

30. There has been disquiet over the use of overseas panel members. The EPSRC argued in 2001 that "the involvement of international expertise is limited so the thoroughness of the international calibration could be questioned".[27] Professor Ivor Crewe said that Universities UK had "no objection in principle to asking those from outside the UK system to judge us in the light of international standards" but in written evidence Universities UK stated that there is a "danger that the inclusion of international researchers [on panels and sub-panels] would be tokenistic, with few international researchers being asked to judge a range of disciplines at panel level".[28] Their concern is that international assessors might not know enough "about the character of the UK higher education system and about the quality of work that particular institutions were submitting to the RAE".[29] Professor Ian Haines is concerned that overseas assessors "are going to have a limited knowledge base and there is a very great danger that they will direct their views in certain areas. If you want to solve that problem, you need a rather large number, and that is likely to be very unwieldy".[30]

31. Sir Gareth Roberts's review reported that is has been acknowledged that in 2001 there was "a weak procedure for using international experts to validate RAE grades".[31] It recommended a "significant international presence on each sub-panel and panel at the point at which it takes its decisions".[32] The Funding Bodies invited "views on how to secure an effective input to the assessment process by people who have direct experience of high-quality research in other countries".[33] In July 2004, the Funding Bodies announced that each main panel would have one or two international assessors but not the sub-panels.[34] We recognised the problems in appointing a large number of international panellists but the Funding Bodies' proposals for one or two assessors on the main panel is unlikely to ease Universities UK's concerns about their tokenistic presence.

32. Sir Gareth Roberts pointed out that the international reviews conducted by the EPSRC could provide a useful source of overseas panel members. In our Report on The Work of the Engineering and Physical Sciences Research Council, we regretted the fact that these reviews were not conducted in the same manner by the other Research Councils to provide a more sophisticated picture of the UK's research competitiveness than that provided by metrics. We agree that the reviewers from the EPSRC's reviews could provide valuable input to future RAEs but we regret that other Research Councils are not undertaking similar exercises on a regular basis. We welcome proposals to strengthen the use of overseas panel members. It should form part of a wider exercise to benchmark UK research.

Workload

33. We expressed concern in our earlier Report about panels' workload. The number of researchers submitted for assessment per panel member ranged from eight to 96 and HEFCE had made it clear in its advice to panel members that "you should not feel that you are required to collect, review or examine all research outputs listed".[35] We noted that the 11 members of the chemistry panel had to sift through over 5,000 submitted outputs. We concluded in our earlier Report that HEFCE had provided inadequate resources to carry out this work. We learnt that panel members had to obtain their own copies of submitted papers.[36] An operational Report of RAE 2001, conducted by the management consultants Universitas and published with Sir Gareth Roberts's Report, was similarly concerned by the workload faced by panels: "The view among panel chairs is that the RAE team was over-stretched and understaffed […] It is highly probable that the team did indeed need more staff during the assessment phase".[37] It goes on to say that "The same degree of dedication and commitment which all those involved showed cannot be assumed for any similar further exercise".[38] The panels and sub-panels need to be properly resourced. Overstretching staff and panel members could lead to panels reviewing individuals selectively and coming up with a biased or wrong conclusion on quality. Under-resourcing is an affront to the researchers and institutions who have gone to the trouble of putting the submissions together.

34. It is clear that the workload of panels is excessive but it is less clear how it can be reduced while all higher education institutions continue to enter the RAE. An excellent opportunity to reduce the burden on panels has been missed by the Funding Bodies in their decision not to support different assessment routes. In particular our proposal to exempt top-ranking departments would reduce the number of submissions that each panel needed to consider and enable them to give closer consideration to submissions.

Assessment criteria

35. For RAE 2001, research was defined in the following terms:

"Research is to be understood as original investigation undertaken in order to gain knowledge and understanding. It includes work of direct relevance to the needs of commerce and industry, as well as to the public and voluntary sectors; scholarship; the invention and generation of ideas, images, performances and artefacts including design, where these lead to new or substantially improved insights; and the use of existing knowledge in experimental development to produce new or substantially improved materials, devices, products and processes, including design and construction. It excludes routine testing and analysis of materials, components and processes, eg for the maintenance of national standards, as distinct from the development of new analytical techniques. It also excludes the development of teaching materials that do not embody original research."[39]

36. Sir Gareth Roberts reported that "There is significant support for a broader definition of research within research assessment, to encompass in particular applied research, research of relevance and utility, training of research students, and research that directly informs teaching". He believed that this "derives from a perception that the RAE has been far too ambiguous about the value of applied research".[40] Richard Lambert also considered the RAE in his Review of Business-University Collaboration, concluding that "World-class excellence across all types of research should be recognised and rewarded by the RAE and Research Council peer review processes. Excellent research undertaken with industry or other users should be recognised as being of equal value to excellent academic research".[41]

37. In response the Funding Bodies announced that they will "ask main panels and sub-panels in all disciplines where this may be an issue to ensure that their criteria statements make clear how they will assess practice-based and applied research, according to criteria reflecting appropriate characteristics of excellence".[42] The Funding Bodies say they are still considering the definition of research to be used for the exercise, but from the starting point that the definition used in 2001 may need to be reviewed rather than changed fundamentally. They have agreed that the 2008 RAE will insist "that panels ensure that criteria are sufficiently flexible that all types of research excellence can be recognised".[43] The Institution of Electrical Engineers remains concerned that "there will continue to be an overemphasis on publications and theoretical work".[44] Professor Ian Haines expressed similar misgivings to us in evidence.[45]

38. In our view the definition used in 2001 covers applied research sufficiently if interpreted correctly. That there is support for a definition of research to be broadened to cover "applied research, research of relevance and utility" indicates that there are problems in the weightings given to applied research by panels. If there is a perception that panels will not give parity to pure and applied research then departments will be disinclined to include applied research outputs in their submissions and ultimately to conduct this research at all.

39. Having defined the breadth of research to be considered, problems inevitably arise as to how the quality of the research outputs is determined. As in 2001, institutions will be asked in 2008 to identify in their submissions up to four pieces of work for each researcher. However, sub-panels may (with the agreement of main panels) elect to set a lower maximum of two or three items where members agree this would be appropriate to a particular unit of assessment.[46] The AUT welcomes the decision to abolish the four items rule and to allow different panels the freedom to define their own limits on the number and size of outputs. Its preference would be for a minimum number of items and for any maximum to be set at a low level.[47] The Wellcome Trust endorses the Funding Bodies' decision to maintain the requirement of four outputs (or fewer) per individual to drive the quality, as opposed to the volume, of research outputs.[48] In some disciplines, research outputs can take years to emerge. We welcome this new flexibility but care must be taken not to set a level too low. If the bar is set too low then too many departments will clear it easily and the basis of the RAE—to allow selective funding—is undermined.

40. Research Councils UK thinks that "contributions made by researchers in promoting and communicating the outcomes of their research should also be recognised as an assessment factor".[49] We believe that academic researchers should devote time to communicating to a wider audience but we have concerns about its use as a criterion of excellence. As a point of principle, the RAE is designed to direct funds to the best researchers and its use to encourage other behaviours is problematic. From a practical point of view, it is difficult to assess the quality of the communication and there is a danger that incorporating it into the RAE would result in poorly conceived and directed activities. We would be interested to learn how the Research Councils might base their funding decisions on the communication skills of their grant applicants.

41. We conclude that the definition of research employed in 2001 is broadly adequate. It is important that the panels give equal weight to pure and applied research and that higher education institutions perceive this to be the case. HEFCE should ensure that it is understood by everyone.

42. A further problem with the RAE has been the use by panels of the place of publication as a proxy for quality. Competition for publication in journals such as Nature is intense and for panels with a large number of outputs to consider it would seem reasonable to conclude that the peer review process conducted by journals did not necessarily need to be repeated. There are two main problems with this approach. Some papers published in Nature fail to achieve a single citation and could be considered to have had negligible impact. As Lord May reported to us, "One of the other perverse consequences you find in common rooms these days as people prepare for these things is that people talk about how many papers in Science and Nature there have been rather than what was in the papers".[50] A researcher could choose to target publication in a highly specialised journal with a small readership drawn from the user community and hence a low impact factor, but which contains results of excellent research. Professor Ian Haines told us that "Some of the greatest break-throughs are to be found in much lesser journals, where people are working in a very specific branch of science and technology. They are often missed".[51] Judgements based solely on the impact factor of the journal could give a highly misleading assessment of the impact and quality of the research. Professor Charles Galasko argues that the "Research Assessment Exercise has been based on naive assumptions, namely that the amount of grant income and the impact factor of the journal in which the work was published is what is important, rather than the impact the research has had". He says that "Research needs to be assessed but perhaps the best way of assessing it would be for a peer group in each discipline to evaluate the impact that the research has had in addition to the potential impact that it may have".[52] It is not acceptable for peer review panels to rely on the place of publication as a guarantee of quality. We recommend that HEFCE to instruct panels to desist from this practice for RAE 2008 and ensure that panels are sufficiently large and well staffed to make informed judgements of the quality of the submissions.

43. Sir Gareth recommended that panels developing their criteria for assessment should be required to ensure that these included suitable criteria for recognising the characteristics of excellence particular to such work. This would be welcome but the problem may stem from the use of the terms national and international excellence since this implies that good research should necessarily have global significance. An alternative system could employ a grading system similar to that employed by some of the Research Councils to rank their grant applications. Table 2 shows the system used by NERC, which concentrates on the potential impact.[53] With modification to reflect the retrospective nature of the RAE, this could be mapped on to the grading profile being adopted for RAE 2008. The RAE should recognise that excellent research may not be internationally significant but it may transform the fortunes of a local business or the provision of public services. We recommend that quality criteria concentrate more on the impact of research rather than the place where it has been published.

Metrics

44. Given the workload of panels, it is reasonable to consider metrics that could provide less onerous but nevertheless reliable assessments of quality. Mr Thirunamachandran outlined to us the three sets of metrics which have been used in the past:

a) Research grant information;

b) Publication information; and

c) Post graduation research unit information.[54]

45. Sir Gareth Roberts told us that he considered the use of metrics to be important in providing the next RAE with "a much lighter touch and less of a burden to both the academics and the assessors".[55] Metrics have two potential applications to research assessment: first as an aid to panels in reaching conclusions; and, second, to replace the peer-review process altogether. We will consider the latter in our discussion of the future of research assessment in paragraphs 67-75 below. This difference is important since the first option would aid the decision-making of panels but not necessarily result in any lightening of the bureaucratic load on institutions unless panels issued clear guidance well in advance. Table 2. Pre-award grading system employed by the Natural Environment Research Council.

Grade Research Strategic Data and Knowledge Shared Services and Facilities Knowledge Transfer

á5

Outstanding: exceptional scientific merit and originality; expected to have major scientific impact; top 5% Outstanding: benchmarks amongst world's best; top 5% of surveys, etc; exceptional delivery, service Outstanding: essential & unique national service or facility; highest quality & scientific standards Outstanding:
Likely to have a major impact on some aspect of improving UK economic competitiveness or effectiveness of public services and policy; top 5%

á4

Excellent: at the forefront of field; will advance understanding; top 25% Excellent: comparable to world leaders in the field; top 25%; excellent delivery, service Excellent: essential and cost effective national service or facility; high quality & scientific standards Excellent:
Likely to have a considerable impact on some aspect of improving UK economic competitiveness or effectiveness of public services and policy; top 25%

á3

Very good: generally competitive science; top 60% Very good: well thought of in the field; top 60%; very good delivery, service Very good: important national service or facility; competitive quality Very Good:
Likely to have a reasonable impact on some aspect of improving UK economic competitiveness or effectiveness of public services and policy; top 60%

á2
Good: quality science, but not leading edge Good: not leading edge; adequate delivery, service Good: useful national service or facility; appropriate quality Good:
Likely to have a modest impact on some aspect of improving UK economic competitiveness or effectiveness of public services and policy

á1
Of merit: modest advance in the field Of merit: satisfactory performance; adequate delivery & service Of merit: sometime useful service or facility; adequate quality Of Merit:
Likely to have a minor impact on some aspect of improving UK economic competitiveness or effectiveness of public services and policy

â Probably not advancing the field; new, useful knowledge N/A N/A Probably not advancing the field.

46. Sir Gareth Roberts assured us that the "research councils and funding councils are really working very, very hard now on appropriate metrics". He expressed confidence that these would be in place for RAE 2008.[56] Mr Thirunamachandran explained that HEFCE was "looking at other metrics, particularly in the science, engineering and technology areas, to see whether there are other metrics which could help us even further, and in the case of applied research and practice based research we think they can".[57] The Funding Bodies said that "Sub-panels will accordingly be encouraged to work with main panels to specify datasets appropriate to their discipline".[58] While we note Lord May's concern over "the babble about metrics", we support the increased use of metrics by panels.[59] It is important that panels provide a clear explanation about how they plan to use metrics as part of their deliberations. We agree with Professor Crewe that "[institutions] will need to know in advance what metrics are going to be used and they would need to be assured that there had been a very thorough appraisal of the validity of these metrics before they were used by the panels".[60] He also insisted that "If a panel wants to know what the quality of an authoritative, historical work is, then there is no substitute for reading a book and the same goes for many of the arts and humanities and social sciences".[61] The Funding Bodies have announced that in August 2005 panels and sub-panels will issue draft working methods and criteria for consultation with a view to publishing their conclusions in November/December 2005. We appreciate that the Funding Bodies only published nomination forms for assessors in July 2004; nevertheless, we consider the timetable to be too long. We recommend that, once formed, panels publish rapidly clear guidance on how they plan to use metrics to inform their appraisals. This should be considered a priority.

Grading system

47. In RAE 2001, 80% of the researchers whose work was assessed were in submissions receiving one of the three top grades, while 55% were included in submissions receiving one of the top two grades (5 and 5*). The amount of discrimination inherent in the exercise is therefore less than the length of the rating scale would suggest. The scale was criticised for the "cliff edges" between grades, where comparatively fine judgements at the grade boundaries could have a disproportionate impact upon funding and reputation. This was also seen as providing a strong incentive for game-playing.[62]

48. Sir Gareth Roberts proposed a quality profile which would set out the proportions of work in each submission reaching each of three defined "starred" quality, and this model has been accepted by the Funding Bodies with the addition of an extra grade (see Table 3).

Table 3. Sample quality profile

Unit of assessment A FTE staff submitted for assessment Percentage of research activity in the submission judged to meet the standard for:

Four star Three star Two star One star Unclassified

University X 50 15 25 40 15 5

University Y 20 0 5 40 45 10

Source: The figures are for fictional universities. They do not indicate expected proportions.

The key advantages of this approach, according to the Funding Bodies, are that it will:

Eliminate the "cliff edge" effects of the previous rating scale.
Put an end to the "averaging" effect of previous RAE grades. It will be possible to distinguish between departments whose work is of even quality and those where the quality is less even, and to highlight the presence of "pockets of excellence".
Put an end to the situation, produced by the previous rating scale, where an HEI might consider leaving one or more established researchers out of a submission to ensure that it achieved a higher grade and thereby attempt to secure more funding.

49. The proposal for a new quality profile has widespread support.[63] The Royal Society proposed a profiling arrangement in its submission to the Roberts Review and pushed for a four star rather than a three star system, arguing that "The profiling system should result in less time and effort spent on 'game-playing' since the reduction in the difference in financial rewards would not warrant it".[64] The AUT describes the new system as "a more sophisticated output than a crude numerical score".[65] The introduction of a quality profile is a significant step forward and, if associated with an equitable funding formula, could eliminate many of the iniquities of the previous grading system.

Playing games

50. The evidence we received during our initial inquiry led us to conclude that "we accept the widespread view that the RAE ratings [in 2001] reflect an improvement in UK higher education research".[66] Nevertheless, for one senior academic the improvement in results represented a "morass of fiddling, finagling and horse trading" and another told us that told us "the results are starting to lack credibility".⁶¹ It was clear to us that universities could play games without any real improvement in research quality.⁶² This could be achieved in several ways:

Exclusion of researchers. Departments were free to enter as many or, more pertinently, as few researchers as they wished. By excluding less productive researchers, the grade awarded could be higher. A fine line needed to be trodden since the amount of funds awarded depended on the number of researchers entered and thus a higher grade could be achieved at the expense of income.
Splitting and merging departments. Researchers could be moved from a weak department to a strong one in such a way as to improve the grade of the weaker one without adversely affecting the grade of the strong one. Also, a 5* department could maintain its grade if merged with a good 4-rated department if some staff were omitted.
Transfers between institutions can distort the RAE results. Transferred researchers need only submit two (rather than four) outputs, thus a researcher's output could be considered to be of higher quality than it really is.

51. The Funding Bodies' acceptance of Gareth Roberts's proposals for a new grading system aim to render ineffective some of these ploys. However, the proposals are generally deemed to have improved the situation but not eliminated it. A problem for universities is that they do not yet know the funding mechanism to be applied in 2008 and they risk devising strategies in a policy vacuum . Somewhat surprisingly, anecdotal reports suggest that this is not proving to be much of a disincentive: Natalie Fenton told us that it was her perception that there had already been an increase in the number of professorships being advertised in an attempt to lure the best researchers.[67] The British Medical Association's Medical Academic Staff Committee Reports that "Many institutions are re-configuring their staffing profiles in anticipation of scoring highly and redundancies are being considered, even before the full detail of the next assessment is published".[68]

52. The stance taken by HEFCE during our earlier inquiry that game-playing was a legitimate part of a department's research strategy was not constructive and we are pleased to hear Sir Howard Newby's less defensive stance: "Yes, it is a problem, and we do recognise it".[69] Lord May told us that "Any system of distributing the money, whether it is the expert peer-reviewed direct costs of grants or the infrastructure costs, anything, other than just giving it out on a per-capita basis, needs rules. And the rules will govern behaviour. There is no way of avoiding it".[70] Nevertheless, the AUT argued that "more could be done to prevent this pernicious form of tactical 'games playing'" and suggested that there should be a "nationally agreed code of practice on research assessment involving the funding councils, Universities UK and the trade unions".[71] A national code of practice has its attractions but it might prove difficult to prove that a university's behaviour was not part of a legitimate strategy. We would prefer the funding bodies to be open and honest about any unintended or unwanted consequences of their policies and methodologies, and identify mechanisms to address them. We welcome HEFCE's acceptance that the tactics employed by universities to improve their RAE grade are not all part of a legitimate research strategy and recommend that it publishes analyses of the strategies being employed by institutions and provide guidelines on what it considers to be acceptable practices

Exclusion of researchers

53. One of the most contentious issues with previous RAEs has been the selective inclusion of researchers, i.e. those termed "research active". The provision allowing the selection of researchers dates from 1992 with the inclusion of the new universities, in which many academics had not previously undertaken research. The post-1992 universities had proportionately fewer academics engaged in research and the selective exclusion of academics enabled the Funding Bodies to direct funds at pockets of research excellence within departments. Nevertheless, this has been seen by many as highly divisive and as undermining the career prospects of those excluded.

54. The advantage of the quality profile is that there should be no financial incentive to exclude researchers; nevertheless, it seems likely that it will still occur. The Biosciences Federation argues that the "quality profile assessment will not discourage the tactical exclusion of weaker researchers" as "many institutions, particularly the stronger ones, will be concerned about the shape of the quality profile and will not want a tail". It says "the effect could be to drive staff into separate research or teaching categories, which may not be in the best interests of undergraduate students".[72] A further likely consequence will be the compilation of league tables which use the "grade point average" as a basis for ranking institutions.[73] In our earlier Report we discussed the use of a high RAE grade as a recruitment tool for undergraduates, arguing that "The best students, particularly from abroad, are likely to be attracted to the universities with the best RAE scores".[74] It seems likely that the media will defy the Funding Bodies' best intentions and distil the quality profile down to a single figure, thus encouraging the exclusion of the least productive researchers if this is permitted.

55. Sir Gareth Roberts told us "I see no reason at all why one cannot submit everybody. It is a dual-support system and to me, for anyone who is eligible for applying to a Research Council for money, it makes sense that they should be included in the RAE".[75] An argument for maintaining the provision to exclude is that there are many academics whose talents lie elsewhere but who would feel obliged to focus more of their energies on research. In past RAEs the identities of those included have not been disclosed, although this may be well known within a department. We would certainly consider it a retrograde step if the status of teaching were further diminished. Sir Howard told us that a decision has not been taken on whether all academics should be included in RAE 2008.[76] We believe that enhancing the status of teaching in universities can best be achieved by complete transparency about the functions performed by different members of the academic staff. To this end, we agree with Sir Gareth Roberts that all academics eligible for Research Council funding are included in a department's RAE submission. We think that greater transparency about the RAE, including the public disclosure of included academics, would have many benefits. It is important to highlight the valuable work done by excellent administrators and by lecturers who invest time and intellect in their teaching. We feel that greater clarity about the role of these academics could increase the value attached to their work. We recommend that as a condition of their block grant, higher education institutions publish an annual staff audit, describing the contributions of all members of academic staff to research, teaching, administrative and other functions.

Women

56. In our earlier Report we expressed concern over the "under-representation of women in the highest-rated departments and that women have been disproportionately excluded from RAE". We welcomed HEFCE's "imminent research project into women in higher education research".[77] Our conclusion was based on HEFCE's 2000 Review of Research Funding, conducted by a committee under the chairmanship of the then Chief Executive of HEFCE, Sir Brian Fender. It recommended that "HEFCE should consider, initially through the human resources sub-group, the relative under-representation of women in the highest-rated departments and whether there are other groups who appear not to be realising their full research potential".[78] HEFCE responded by saying that "We have to be as certain as we can be that the RAE processes do not unwittingly give rise to unwarranted discrimination".[79] Despite this positive response, to our knowledge no research has been published by HEFCE. The issue has remained a live one, however, and in June 2004, the AUT published a Report which provided detailed figures of the representation of women in RAE submissions.[80] The study fails to provide evidence of a causal link between the RAE and women's academic careers. The Wellcome Trust has commissioned research which showed that, while women are as successful with research grant applications, they are less likely to apply.[81] Whatever the causes for this, it is reasonable to assume that fewer research grants would lead to fewer research outputs (publications) and would therefore mean that women would be less likely to included in the RAE. In our earlier Report we reported that women academics were more likely to take on more of the teaching and pastoral functions within departments. It is our view that the issue to be resolved here is the status accorded to academics who take on these non-research but nonetheless essential roles.

Confidentiality

57. The Funding Bodies say that "the RAE process will not provide for either sub-panels or main panels to reach collective judgements on the overall quality of a named individual's work and outputs, so no such collective judgements will be available to be disclosed".[82] In the new system sub-panels will be asked to produce a profile which reflects all of the information contained within a submission. The submission will contain a number of research outputs which can be linked to individuals, and other information which cannot be linked to individuals (metrics and strategy).[83] The emphasis that panels place on each of these data will vary but it seems likely that publications by individuals will be given greater weight. This prompts the UK Computing Research Committee to express concern that "if the position [confidentiality] will be able to be maintained in the face of challenge under the Freedom of Information Act, the Data Protection Act and the Human Rights legislation. […] the effects of individual ratings becoming known may do more harm to research, through undermining morale and creating divisions inside research teams and departments, than the rating process does good in raising research quality".[84] We do not share this concern. Academics know, or think they know, who the best performers are. We argue above that there should be disclosure of the researchers included for assessment and greater clarity of the role that individual academics play within their department. It would be a logical extension to publish an independent appraisal of academics' research records. HEFCE assures us that panel members, secretaries and RAE team staff will be bound by a duty of confidentiality. We anticipate that this will be challenged in the courts in a bid to reveal publicly the judgements made about the performance of researchers. We recommend that such a move should be pre-empted and that the grades awarded to individual researchers should be made public. This would bring welcome transparency to the process.

Research competence

58. Concern has been expressed about the effect of the RAE on the careers of those not entered for the exercise. In our earlier Report we concluded that being labelled "research inactive" for tactical reasons can blight research careers, and even bring them to an end. We also heard concerns that the RAE may have contributed to the large increase in fixed term research positions.[85] In RAE 2001, departments' submissions included a statement of the unit's research strategy and environment. The idea was that this would inform panels' decisions on grades. Sir Gareth described this approach as only "partially effective" since panels tended to concentrate on research outputs.[86] The operational report for RAE 2001 also notes that "Panel chairs were split between those for whom the textual commentary in forms RA5 and RA6 [which covered staffing issues and the research environment] was important or essential, and those for whom it was of little or marginal use".[87] Sir Gareth felt that research competence needed to be separated out from any deliberations of research quality and that failure to meet agreed standards should result in some sort of penalty. He proposed that institutions should demonstrate "research competence", separate from the assessment of research quality. He suggested four main criteria:

a) research strategy (the coherence of an institution's research strategy including an assessment of the credibility of its targets for obtaining funding);

b) development of researchers, including postgraduate research students, postdoctoral researchers and junior lecturers;

c) equal opportunities policies and success in putting them into practice (this would relate to an institution's policies for ensuring equality of opportunity for all its staff, not just those in research roles); and

d) dissemination of research beyond the academic peer group. This would cover an institution's policy on encouraging a spectrum of activities, ranging from collaboration with organisations outside HE, through the use of research to enhance teaching, and work promoting the public understanding of research topics.

59. Sir Gareth proposed that an institution failing its assessment against any one of the competences would be allowed to enter the next research assessment, but would not receive funding on the basis of its performance in that assessment until it had demonstrated a satisfactory performance.[88]

60. Research Councils UK also wishes to see broadening of research criteria, to include:

a) a collaborative team-based approach to research;

b) multidisciplinary research;

c) a proactive relationship with research users;

d) the provision of high quality research training;

e) research governance; and

f) knowledge transfer.

Sir Gareth's proposals were rejected by the Funding Bodies because of concerns that it would be "unduly complex and could impose a greater burden than the likely outcomes would justify".[89] Instead departmental research strategies will be assessed through the RAE process as in previous exercises. Issues of equal opportunities, staff development and dissemination may also be addressed by the Funding Bodies through existing mechanisms outside the research assessment process.[90] Professor Crewe pointed out that the funds to promote these activities amounted to a "substantial tranche of money".[91] Research Councils UK says it is disappointed that the assessment of institutional competences has been abandoned.[92]

61. A longstanding concern of ours has been the viability of research careers and the treatment of contract researchers.[93] Thus we also sympathise with attempts to address wider problems in the research environment. Nevertheless, we think the Funding Bodies' decision was the right one. If there is a demand to use financial levers to correct behaviours in higher education, then the Funding Bodies have the option of broadening an existing mechanism such as the RAE or introducing new ones. In our earlier Report, we concluded that while the RAE had its faults, many of the wider impacts attributed to it resulted from the absence of suitable financial incentives in other aspects of universities' missions. Natalie Fenton, in supporting Sir Gareth's proposals, told us that "There should be all sorts of other factors that will have a play on the research standing of an institution, which should be taken account of within the RAE".[94] Although Sir Gareth's proposals for the assessment of "research competence", if implemented, would place a heavier burden on the RAE, we believe that the need to promote good practice is so important that it should be use alongside other incentives to promote good practice in higher education research.

Frequency

62. After RAE 2008, the Funding Bodies plan to work to a six-year assessment cycle. This is consistent with our recommendation of a six-yearly cycle in phase with the Spending Reviews.[95] Sir Gareth recommended that there should be light-touch "mid-point monitoring". This would be designed only to highlight significant changes in the volume of activity in each unit.[96] This was rejected by the Funding Bodies on the basis that it was unduly complex and could impose a greater burden than the likely outcomes would justify. The Royal Society and Universities UK support the decision.[97] The Institute of Physics is disappointed by this rejection, however, as is the British Society of Criminology.[98] We concluded in our earlier report that departments could apply for regrading to recognise the effects of increased investment, thus providing only a self-imposed burden. We are disappointed that the Funding Bodies have rejected any form of mid-point monitoring. The RAE is designed to fund research excellence selectively and this funding should therefore reflect a department's current, and not only past, capabilities.

Costs

63. There is an important distinction between the funds invested by HEFCE to ensure that the process runs smoothly and effectively and the costs imposed on institutions. We discussed the panels' workload above in paragraphs 33-34 and argued that they needed greater financial support. Concerning the costs to institutions, HEFCE told us that "The [indirect] cost of the 1996 RAE has been estimated at between £30 million and £37m and it is likely that the costs of RAE 2008 will exceed this". It comments that this "represents not much over 1% of the resources to be distributed by the funding bodies in the period 2002-03 to 2008-09 using the ratings. This is well below the proportionate cost of research grant allocation systems based upon bidding for projects". It insists that "Research quality assessment is an essential tool to provide assurance that the considerable public funds spent on research in HEIs are being put to good use" and that "much of the activity identified in our latest costing study would have been undertaken anyway by well-managed institutions planning and reviewing their research effort".[99]

64. The Funding Bodies proposals state that "In developing our plans for the next RAE we have paid particular attention to ensuring that the cost of the exercise, and the administrative burden that it will place upon HEIs, are kept to the minimum—having regard to its expected impact and to the resources to be allocated using its outcomes". The Funding Bodies calculate that the direct costs of the 2001 exercise to the Funding Bodies came to £5.6 million but that the figure for 2008 is expected to be £10 million.[100]

65. While Universities UK is "reassured that the sector's concerns about the level of bureaucracy imposed by the current system—and potentially by the Roberts proposals—seem to have been taken on board in the revised proposals for a single assessment process", the Biosciences Federation Reports that "opinion is divided on whether the outcome justifies the burden and cost involved". [101]

66. The figures provided by the Funding Councils of the cost of the RAE to institutions do not appear to be excessive. The fact remains that this burden is resented by universities. The Funding Bodies should be sensitive to this feeling when developing their plans for 2008.

Conclusion

67. There has been a general acceptance of many of the decisions made by the Funding Bodies for RAE 2008. Universities UK has "broadly welcomed the announcement of the Initial Decisions by the Funding Councils" and the Royal Society "believe[s] that the proposals are a significant step forward"^.[102] Many of the concerns stem from the fact that there had been only an "initial statement"; there was, in Universities UK words, a "lack of detail available on many key aspects of the proposals".[103] The Association of University Teachers (AUT) had similar worries: "Key decisions such as panel criteria and crucially, levels of funding, have yet to be decided so it is difficult to make a definite judgement at this stage".[104] Research Councils UK also felt that "there appears to have been some shying away from the introduction of radical reforms".[105] The Funding Bodies' proposals have addressed positively many of our concerns about the RAE mechanism and HEFCE has adopted a more open-minded and constructive approach to its reform, which is a welcome change. A more radical approach, employing a range of metrics to reduce the bureaucratic burden on universities is still needed. We accept that their application will be a complex and time-consuming task for RAE and the Funding Bodies but we believe that the administrative burden should fall here rather than on the universities.