Examination of Witnesses (Questions 80-99)|
WEDNESDAY 16 OCTOBER
80. I am not at all sure what the original inquiry
wanted to know. You do say in the report that there was a concern
about systematic drift and I think this Committee would like to
have some knowledge about how we ascertain systematic drift and
how we do anything about it and how we are going to respond to
(Sir John Kingman) Exactly so. There is a lot of research,
a lot of academics have been in touch with us with interest about
this. There is a lot of educational research. It is not fundamentally
a statistical question. Our concern has been to make sure that
the results of this academic research are available to and used
properly by the policy makers in the department. As far as we
can tell that is happening but it is quite difficult to be sure
of that. You cannot ask the Statistics Commission whether there
is grade drift and so on, those are not statistical questions,
they are educational questions which are extremely difficult.
They can sometimes have a statistical component but it would be
wrong for the Statistics Commission to pontificate about that.
81. The next question I wanted to ask might
perhaps not be for you. Of the statistical component would you
say that it was significant enough as to be of concern or will
I find that out from the note?
(Sir John Kingman) I do not think I can really help
on that, I am sorry. I think you are going outside the proper
sphere of the Statistics Commission.
82. In that case I will wait and see your note.
(Sir John Kingman) I wrote to Len Cook on 1 October,
I wrote several letters to Len Cook on 1 October, and noted that
there was a report in 1999 which investigated accusations that
test standards were being deliberately manipulated and concluded
that there was no evidence of that, but I went on to say that
we know that the Office for National Statistics is having a quality
review on education performance statistics starting sometime next
year and we are really encouraging him and the department to take
this seriously. "I would be grateful if you could ensure
that this area is examined in the near future and let me know
what the timescale will be."
83. Some of the questions I would have liked
to pursue this afternoon perhaps were to do with how the statistics
are compiled in the first place and what confidence we can have
in them but if you are not ready to deal with that then I will
have to be patient.
(Sir John Kingman) I would like to offer you a note
that summarises the various lines of correspondence that have
happened already and obviously this letter will be a part of that.
84. Can we move on to hospital waiting lists.
You say that you are satisfied that there has not been any interference
in the statistics and the presentation of the information about
hospital waiting lists and that you are satisfied that this has
all been done in the usual way. Can you describe to me what the
"usual" way is. I will tell you why I am asking you
that. When I was in my local hospital recently a surgeon said
to me, "Look at those figures up there. They are all wrong,
you know, I have not put mine in." What confidence can I
have as a politician, if a surgeon can tell me that he has not
put his figures in yet, that what I am getting at the end of the
process is an accurate picture of what is happening?
(Sir John Kingman) We do not have the resources to
go hospital by hospital and check what is going on and what we
are trying to do is to look at the ways in which the hospital
statistics are being put together, the checks on them and so on,
and also the work that the National Audit Office, which does have
the resources, is doing on hospital waiting lists. There is obviously
a possible overlap between the interests of the Commission and
the interests of the National Audit Office because they are concerned
that when these numbers are put out they do actually contain everyone's
figures and there are not miscountings or deliberate deceptions
about the figures. And it is certainly suspicious that there are
still substantial seasonal variations in the waiting list which
is linked to the dates when the performance statistics are produced.
So there is room for concern and what we are trying to do is to
work with the National Audit Office so that their big battalians
can look at the sort of questions which worry us about the validity
of the statistics themselves. That is something where the National
Statistician also has an important interest, so the three of us
are trying to work together.
85. So the dip that occurred before the General
Election, which I assume is what the media were looking at which
prompted your interest, presumably from some concern about interference
(and I may be wrong in that) if that was a seasonal dip that we
might expect periodically, by this amount of time would we have
expected to have seen yet another seasonal dip? If so, have we
(Sir John Kingman) We have seen seasonal dips each
year and we have not had an election each year, so it is clearly
not just an election phenomenon.
86. That is good to know.
(Sir John Kingman) It may be a phenomenonand
I only say "may be" because this is still very much
under studyto do with the dates that affect people's performance
pay and that sort of thing. These are the sorts of hypotheses
which the National Audit Office and we are interested in.
87. Would you pursue evidence to support that
(Sir John Kingman) I think if there were any serious
suspicions that that sort of thing was happening, that is something
we would be very interested in.
88. As it is, you are assured it is perfectly
(Sir John Kingman) No, I did not say that.
89. What are you saying?
(Sir John Kingman) What I am saying is we and the
National Audit Office are working on it and that they are the
people who have the big batallions that can investigate this at
the hospital level.
90. So your satisfaction only lies in the fact
that the statistics were compiled in the usual way not that the
the statistics themselves are satisfactory?
(Sir John Kingman) That is right.
91. Just following up this particular area,
I am pleased that you said earlier that you wanted to focus as
a group on some of the key statistics that are of particular interest
to people in the country and raise confidence about those statistics.
We discussed this issue of the waiting lists and whether there
was anything suspicions in the pattern. One of the issues, as
you know, that is raised by the waiting lists is those treatments
and types of medical wait that are excluded from them. Many of
us see in our surgeries the whole time people who in many cases
have a two-year wait to see a consultant and you discover when
you go to the hospital that they are not counted as being on a
waiting list that is looked at nationally. Consultant referrals,
diagnostic tests, angiograms, MRI scans and endoscopies are not
on that. Is that the type of issue which you as a group can be
concerned about, the quality and the extent of the data, particularly
as you want to reinforce its public credibility?
(Sir John Kingman) This is a very important area because
it is clear that the system of targets has come to stay in our
political life. The Government will lay down targets and will
tie Treasury funding to the achievement or otherwise of those
targets and inevitably choosing particular targets which will
be defined simply enough for people to understand will distort
the service. Whatever you choose people will play to those targets
rather than to other things which have not been included in the
targets. The temptation will then be to have more and more targets
and many of these will be very difficult to measure.
92. But do you see it as part of your job to
improve the quality of statistics such as these, for example,
by including people in the statistics who are waiting for primary
care treatments, by including people who are waiting consultant
to consultant, including people who are waiting for angiograms,
diagnostic testing? Do you think the credibility of these statistics
about waiting times in the NHS is important, that it is a legitimate
area of inquiry for your group to look at whether the statistics
should be widened and amended?
(Sir John Kingman) I think our main function, our
core function, is to ensure that where a target has been laid
down the statistics which decide whether that target has been
met should be statistics in which people can have confidence.
That has two aspects. It involves ensuring that the targets that
are laid down are actually measurable and, secondly, ensuring,
as with all government statistics, that the statistical outputs
are high quality and not distorted. You are asking us whether
we would want to move into a different area of actually criticising
the targets themselves. Strictly speaking I do not think that
is part of our terms of reference although it is almost impossible
to exclude that sort of consideration when you start looking at
93. So if people raise that issue with you,
if my constituent who is excluded from all these waiting lists,
both inpatient and outpatient, and is waiting until August 2004
for a neurophysiology report, came to you and said "This
is ridiculous, the Government is highlighting all these areas
that are publicly visible but I am being left to wait for this
consultant to consultant appointment", is that something
as a broad issue that you as a group would look at?
(Sir John Kingman) It is something about which every
citizen would have concern. I do not know whether the Statistics
Commission would be regarded as having a particular expertise
in that area that would influence ministers. After all, the choice
of targets is typically a highly political matter and we can give
advice but whether our advice would be thought to have any value
in that sort of case I do not know.
94. But you could give advice?
(Sir John Kingman) We can give advice on anything
but we are better to concentrate on those areas where we have
some demonstrable expertise.
95. May I ask you then very briefly whether
you would comment on two other things in relation to this same
issue of hospital statistics. The first is that there are some
hospital statistics about particular waiting times that may be
kept by hospitals but are not collected centrally and reported
nationally and they would be in the types of areas I have discussed.
I will ask you the other question to save time. Is it the job
of the Statistics Commission to comment on that sort of thing
about the case for publishing that type of data nationally? In
terms of some of these areas of waiting times there seems to be
an inconsistency in the way that different hospital trusts record
data. In a newspaper article earlier on this year about this type
of problem they were talking about the way in which some trusts
count endoscopies as part of the waiting list and some do not.
They reported a letter to chief executives of trusts in the South
East of England from the NHS Regional Director advising that endoscopies
should be removed from the waiting list but saying that this should
be done gradually to avoid some lurches in the figures. Is that
something that would be of concern to the Statistics Commission
that you might comment on?
(Sir John Kingman) I have not seen that letter but
that sounds to me a letter that would concern us because if statistical
information is to be of any value it must be the real information
and not information which is tapered in order to avoid lurches.
I have not seen the letter. If I do, I think it is something the
Commission ought to consider.
96. If I can get you a copy, it is something
you will look at?
(Sir John Kingman) Certainly. We are charged with
looking at the scope of National Statistics and there are some
very curious border lines in the National Health Service area
as to which figures are National Statistics and which are not.
We have not done a lot of work so far on the scope of National
Statistics simply because it was pointless to do so before we
had the Code of Practice which said, effectively, what it means
to be a national statistic. Now that we have that Code, there
are a number of areas of which I think the Health Service will
be one where we will want to look at the boundary between National
Statistics and management information which is not National Statistics.
Clearly every organisation collects numerical information, some
of it of a very trivial sort which it would be ridiculous to put
within the discipline of National Statistics, but where exactly
you draw the line does need careful consideration and the sort
of example you cite shows how important this is in the NHS area.
97. Finally, could we turn to the Census on
which you have commented in your report. The first Census result
seemed to show that the population in the UK was about a million
less than previously forecast which seemed to come as a bit of
surprise to me. Was it a surprise to you?
(Sir John Kingman) Yes, I think it was a surprise
98. Does that suggest there are some problems
with the methodology of the Census or the methods used on the
(Sir John Kingman) It certainly produces very important
questions that need to be answered. If a scientist measures the
same thing in two different ways and gets quite different answers
he should not say which is right; he should say what is the reason
for the discrepancy and let us try and understand it. On the face
of it, the Census results ought to be more accurate than the extrapolations
that are done between Censuses. If that turns out to be the case
when there has been more detailed study that will not be at all
surprising and it will be a caution to us about using the extrapolation
techniques. This, of course, has very important implications for
the question of what the next Census, if any, should look like
because clearly if our methods of looking at births and deaths
and migration are not sufficiently accurate to give us a figure
within a million, a true figure, then they should not be used
over a 10-year period to make important policy decisions. But
I think there are still a lot of questions to be answered about
this last Census in relation to the people who may have been missed
both by the Census itself and by the subsequent Census coverage
survey. The estimates about missing people are normally taken
from an assumption that there is independence between these two
exercises. In fact, there are likely to be particular categories
of people who are missed out both by the Census and by the coverage
survey and whether that is part of the explanation of the million
difference between the two figures is still a question for discussion.
99. You have anticipated my second question
on the Census. It is perhaps important to distinguish between
the two different figures of a million, is it not, the million
lower overall population and then the adjustable figure of around
about a million to take care of people who were not there? Would
you normally have expected that adjustment of a million to be
of that order from the coverage survey?
(Sir John Kingman) Yes, the fact that you did not
get Census returns from a million people is not at all surprising
and is in line with international experience. The techniques for
what they call "imputation", although they are very
easy to make fun of in the press, are in fact very well grounded
in statistical principle. It is important to realise that the
estimate of the total population size that results is a statistical
estimate. The experts have estimated that it is a figure plus
or minus 100,000, so that needs to be borne in mind in using the
figures. Of course when you break those figures down to smaller
areas you get larger proportional errors. That is all within the
scope of standard statistical methodology, there is nothing to
worry about there. I think the discrepancy between the estimate
that arises from the Census and the estimates that have been made
over the 10 years since the last Census do need very careful study.
I suspect that some of them will turn out to result from mistakes,
not mistakes but poor methodology in 1991 which formed the base
of the extrapolations. If those were then, of course, the extrapolations
from them are bound to be wrong. I think it is important, and
ONS agree with this, to understand where these discrepancies come
from and, of course, to break them down at the local level as
well because they do seem to be concentrated in certain areas
where there is particular difficulty.