Memorandum from the Royal Statistical
Society (HE 36)
THE QAA ASSESSMENT
ON HE INSTITUTIONS
1.1 The Royal Statistical Society welcomes
the opportunity to comment on the effect of QAA assessment in
its present and proposed forms. The Society recognises in the
context of a publicly funded HE system, the need for a method
of reporting which both identifies unsatisfactory practice and
acknowledges excellence. However, we would wish to make a number
of points concerning the details of the methodology adopted.
2. THE SYSTEM
2.1 The present system of grading the six
aspects of assessment, on a scale from 1 (worst) to 4 (best) invites
aggregation by institutions to give an "overall score".
However, scores formed from the sum of six separate scores, each
of which covers a very narrow scale, are likely to be very "bunched"
and consequently lacking in discriminatory power. This has indeed
proved to be the case, particularly in the later stages of the
present review cycle, with the great majority of scores awarded
falling in a narrow band from about 19 to 24.
2.2 The undesirable result of this is that
a score made up of three 4's and three 3'stotalling 21has
in some institutions come to be regarded as "unsatisfactory"
or "a cause for concern", even though such a profile
could indicate no major areas of deficiency. Conversely, a total
of 21 could also be generated by a profile (admittedly unlikely)
of five 4's and a 1, thus masking a major problem. Moreover, any
"league tables" derived from such a system are fairly
meaningless, as has been amply demonstrated already in the case
of schools: a one-point change in one score for one subject area
can result in a drastic change in an institution's "league"
2.3 Under the revised methodology currently
being piloted by the QAA, the use of numerical scores will be
abandoned. However, it will be replaced by grading of three aspects
of provision on a 3-point scale, "failing", "approved",
and "commendable", with a fourth category, "exemplary",
also available in exceptional cases. This is effectively a four-point
scale by any other name, and once again institutionsand
perhaps more importantly the media with their fondness for creating
"league tables"will almost certainly succumb
to the temptation to allocate numerical equivalents to the verbal
assessment grades, and then proceed much as at present.
2.4 The basic problem here is the need to
measure a very complex entitythe "quality" of
the educational experience offered by an institution/experienced
by a student. At present this is being done by the use of small
number of parameters on a restricted scale, allocated via a system
which, however carefully specified and regulated, will inevitably
contain a high degree of subjectivity. This is no problem so long
as the results are properly understood in the context in which
they are derived; however, once allocated the "measures"
tend to assume a life of their own, and to be regarded as having
a validity which they certainly do not possess. In terms of repeatability,
for example, it is unlikely that a different team of assessors,
visiting the same area of an institution at a different point
in time, would generate precisely the same profile of results.
2.5 However, we would recommend the adoption
either of a purely verbal reporting system, or of a much more
detailed numerical system which will better discriminate between
units and not have such an inbuilt tendency to generate very similar
3. GENERAL OBSERVATIONS
3.1 It is a well established fact that in
many commercial and industrial situations, the establishment of
targets or systems of measurement results in staff "working
to the target", with a consequent tendency to neglect areas
which are not being measured. Initiatives which, while possibly
beneficial in the long-term, might produce short-term deteriorations
in the measured results also tend to be stifled. This is not a
state of affairs which would be beneficial to higher education
in the long run. In designing any quality measurement system,
therefore, great attention needs to be paid to questions such
"For whose benefit are these
measures intended?" (for example, the views of students and
of Government ministers as to the relative importance of the various
aspects being assessed would probably be quite different);
"Are we measuring what we think
we are measuring?"
"Are the measurements actually
being changed by the measurement process?"
"Are we measuring the things
which really need measuring?"
3.2 Much expertise has been developed in
the commercial context to deal with these issues, and Fellows
of the Royal Statistical Society working in the quality improvement
field have played a major part in these developments. It would
be unfortunate if the current development of quality measurement
systems in an HE context failed to build on what has been achieved
elsewhere, or repeated easily avoidable mistakes.
3.3 The Society would be happy to provide
further input to the Inquiry on these issues.
Royal Statistical Society