Select Committee on Education and Employment Appendices to the Minutes of Evidence


Memorandum from the Royal Statistical Society (HE 36)


  1.1  The Royal Statistical Society welcomes the opportunity to comment on the effect of QAA assessment in its present and proposed forms. The Society recognises in the context of a publicly funded HE system, the need for a method of reporting which both identifies unsatisfactory practice and acknowledges excellence. However, we would wish to make a number of points concerning the details of the methodology adopted.


  2.1  The present system of grading the six aspects of assessment, on a scale from 1 (worst) to 4 (best) invites aggregation by institutions to give an "overall score". However, scores formed from the sum of six separate scores, each of which covers a very narrow scale, are likely to be very "bunched" and consequently lacking in discriminatory power. This has indeed proved to be the case, particularly in the later stages of the present review cycle, with the great majority of scores awarded falling in a narrow band from about 19 to 24.

  2.2  The undesirable result of this is that a score made up of three 4's and three 3's—totalling 21—has in some institutions come to be regarded as "unsatisfactory" or "a cause for concern", even though such a profile could indicate no major areas of deficiency. Conversely, a total of 21 could also be generated by a profile (admittedly unlikely) of five 4's and a 1, thus masking a major problem. Moreover, any "league tables" derived from such a system are fairly meaningless, as has been amply demonstrated already in the case of schools: a one-point change in one score for one subject area can result in a drastic change in an institution's "league" position.

  2.3  Under the revised methodology currently being piloted by the QAA, the use of numerical scores will be abandoned. However, it will be replaced by grading of three aspects of provision on a 3-point scale, "failing", "approved", and "commendable", with a fourth category, "exemplary", also available in exceptional cases. This is effectively a four-point scale by any other name, and once again institutions—and perhaps more importantly the media with their fondness for creating "league tables"—will almost certainly succumb to the temptation to allocate numerical equivalents to the verbal assessment grades, and then proceed much as at present.

  2.4  The basic problem here is the need to measure a very complex entity—the "quality" of the educational experience offered by an institution/experienced by a student. At present this is being done by the use of small number of parameters on a restricted scale, allocated via a system which, however carefully specified and regulated, will inevitably contain a high degree of subjectivity. This is no problem so long as the results are properly understood in the context in which they are derived; however, once allocated the "measures" tend to assume a life of their own, and to be regarded as having a validity which they certainly do not possess. In terms of repeatability, for example, it is unlikely that a different team of assessors, visiting the same area of an institution at a different point in time, would generate precisely the same profile of results.

  2.5  However, we would recommend the adoption either of a purely verbal reporting system, or of a much more detailed numerical system which will better discriminate between units and not have such an inbuilt tendency to generate very similar scores.


  3.1  It is a well established fact that in many commercial and industrial situations, the establishment of targets or systems of measurement results in staff "working to the target", with a consequent tendency to neglect areas which are not being measured. Initiatives which, while possibly beneficial in the long-term, might produce short-term deteriorations in the measured results also tend to be stifled. This is not a state of affairs which would be beneficial to higher education in the long run. In designing any quality measurement system, therefore, great attention needs to be paid to questions such as:

    —  "For whose benefit are these measures intended?" (for example, the views of students and of Government ministers as to the relative importance of the various aspects being assessed would probably be quite different);

    —  "Are we measuring what we think we are measuring?"

    —  "Are the measurements actually being changed by the measurement process?"

    —  "Are we measuring the things which really need measuring?"

  3.2  Much expertise has been developed in the commercial context to deal with these issues, and Fellows of the Royal Statistical Society working in the quality improvement field have played a major part in these developments. It would be unfortunate if the current development of quality measurement systems in an HE context failed to build on what has been achieved elsewhere, or repeated easily avoidable mistakes.

  3.3  The Society would be happy to provide further input to the Inquiry on these issues.

Royal Statistical Society

January 2000

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2001
Prepared 23 March 2001