Examination of Witnesses (Questions 40
TUESDAY 9 JULY 2002
40. And also, if the running costsif
you were not contracting out that work, presumably you would then
argue that your human resources budget would increase?
(Sir John Bourn) Yes.
41. The 10 per cent budget increase will allow
the production of 60 reports, as against 50 Value for Money reports
previously. Has the National Audit Office considered the trade-off
between quality and quantity in the number of Value for Money
reports it produces and might it not be better to improve the
effectiveness of the existing 50 reports each year, for example,
by revisiting to check that improvements have really been made,
than to produce 60 reports in future?
(Sir John Bourn) Well, within the 60 reports, currently
we have 14 reports in progress which do involve checking up and
following up previous reports, PAC decisions and government acceptances.
Three examples would be: we are doing a follow-up report on central
government construction; a follow-up report on hip replacements;
a follow-up report on hospital-acquired infection. So we do, within
that 60, have quite a lot of follow-up. It is an important point
that we should, in our reports, where this is appropriate and
feasible, have a section on following up. We now provide in each
of the Value for Money reports, references to previous reports
in this areawhat the PAC had recommended and what the government
have come back to. So, in going for 60, that is not at the expense
of neglecting follow-up. Follow-up is provided for.
42. Would a report in itself suggest the need
(Sir John Bourn) Well, sometimes it may well do. I
think that the team that is working on a subject may, as they
do the work, see the connection between their present activity
and some allied activity and therefore bring into the study some
follow up elements.
43. Thank you. The NAO Corporate Plan mentions,
on a number of occasions, its target of saving the taxpayer eight
times its net cost. That is a bit like the effectiveness of Customs
and Excise, and one gets the impression that the more money that
is put in, the greater the savings would be. If the NAO was really
saving the taxpayer eight times its net cost and this relationship
would also hold for budget increases, the 10 per cent increase
is clearly a good deal for the taxpayer. Could you provide more
information to the Commission on these calculations and the extent
to which they represent real resource savings to the economy?
(Sir John Bourn) Yes, I am certainly glad to develop
that point, because they are savings which come from our recommendations
when endorsed by the Committee of Public Accounts, when accepted
by the Government in the Treasury Minute, then costed out by the
Department itself in conjunction with us. But it is not us saying
"This is our view of what could be saved", this is the
Department concerned saying "Yes, this is what the savings
would be". And we have that looked at, and the methodology
of that is looked at, by our internal auditors, who, going back
to a previous discussion, are in fact an independent set of people,
not employees of the NAO, reporting to the chairman of the Audit
Committee, who, again, is not an NAO person. So, there is a proper
methodology for doing this, and it means that the savings figures
that we talk about are agreed by the Department and do represent
real savings. Examples of the kind of savings that they represent
is if a report points out that a Department owns more land and
property than it is using, there are recommendations to dispose
of it, sell it, lease it out.
Another area which has produced a lot of savings
is making people more sensitive to what they are paying for public
utilities: gas, water, electricity and the telephones. Things
you would think they might do themselves, but often are sadly
neglected. In a number of cases it links up with your previous
question. Departments are often bad at chasing up people who owe
them money, and that is something that can be underlined. There
are a whole series of areas in which these savings accrue but,
it is not just us saying so, it is with the Department and subject
to being looked at by a separate group of people who are not our
44. One of the things you always tell us to
do on the Public Accounts Committee is see whether the civil servants
whose projects go pear-shaped underneath them, actually bear the
personal cost of that, ie their promotion is held back, or they
do not progress in the Department itself. The LSE does these quite
useful reviews of your Value for Money reports. I was wondering
whether National Audit Office teams who produce not very good
reportsaccording to the LSE or indeed in your own judgmentpay
some sort of consequence in terms of their promotion and prospects?
(Sir John Bourn) Yes, they do: they get less chance
of getting promoted.
45. And you are quite tough about that?
(Sir John Bourn) Yes. I want to encourage them to
do better so we will provide training, and we have a whole system
of encouraging and helping people to be better. But certainly,
those who do not respond do not do so well, both in terms of their
earningsthey fall behind their peersand certainly
as far as promotion is concerned, they do not get it.
46. And these LSE external evaluation reports
are quite an important part of that process, or not?
(Sir John Bourn) Yes, they are. This is a system of
evaluation that we introduced ourselves. It had seemed to us that
among the various ways in which you had your reports evaluated,
it would be quite good to get a university of intelligent people
coming at it from a slightly different point of view, and, first
of all, Brunel did itthey were the people who won the first
contract to do itand when that was up we had a competition
and LSE won it. And it is taken as an important matter, and it
is interesting how these things work in organisations. The people
in the NAO, above all, are concerned with what the PAC make of
it and whether it goes down well with the committee; and also
whether the government accept the recommendations. But it has
been interesting to see the desire to get your ratings from LSE
up as well. So, it has worked as a professional pride incentive
to try and do better and show these people, coming at it, as I
say, from a rather different point of view, that the teams can
47. Is it also used as an evaluation of their
(Sir John Bourn) Yes, it is.
48. I think other members of the Commission
are going to talk more about, for example, the report on HM Prison
Service. Although the scores from the LSE team were all right,
there were some quite strong criticisms of the methodology, for
example. I was wondering what happened to the team who produced
this Value for Money report. Were they given more training; were
they held back; have they been promoted; what has happened?
(Sir John Bourn) They certainly were given the opportunity
to have more training. I cannot recall off-hand all the names
of the people concerned; but certainly, in the assessment of the
annual performance and what that translates into money; on the
Value for Money side, if you have not come up with a sensible
methodology, if you have not shown some enterprise and ingenuity
in developing it and gone out and found people who have an interesting
line that can realistically be taken, you will fall back.
49. Following on with that thought, looking
at your own notes which you have prepared for us on the trend
analysis of the LSE scores awarded, there is this graph, and this
shows the percentages of reports scoring three or more, and three,
I think, is recorded as "solid professional performance".
How many of them fall below "solid professional performance",
and how many above? Do you have that sort of information?
(Mr Whitehouse) In the last financial year, 43 per
cent of the NAO's reports scored a four or a five overall, which
is very good or excellent/outstanding, using our categories. The
remainder were three. We had no report that scored an overall
two, as "below a professional performance".
50. Looking at the overall column, which is
the last column, and if you look at the dotted line which is 1998,
and the solid black line which is 2000-01; in that final column
the 1998 figure was over 10 per cent below the 2001 figure. What
shortcomings did you have to put right to get those overall improvements?
(Sir John Bourn) That did lead us to take various
measures, which have had an impact on the scores in subsequent
years, to improve the training in relation to the methodology,
and that has taken a number of forms, in terms of the study guides
produced by the Value for Money Development Team. It has also
taken the form of a series of what we call "master classes",
where we invite people to come to the office who have some methodological
sophistication which seems relevant. We had, for example, the
recent Cambridge Professor who won the Nobel Prize in Economics
come to the office to give his views on how we could take account
of his insight. As well as that, we have introduced with LSE in
some cases not only the sort of cold reviews that come at the
end, but hot reviews while the study is going on, so that LSE
people could say before the study is set in stone, "What
about your methodology in relation to this or that matter?"
I think some of the recent reports do bring that out. For example,
the report that we did on the government's gold sales was informed
by Professor Binmore's expertise on auction theory. The report
we have just finished on helicopter logistics has used a variety
of operational research techniques relating to supply chain management.
We have used aspects of economic modelling in the work we did
on New Deal for Young People and what that meant in terms of employment
created for them. So, it was an important point that LSE made.
It is something that we recognise and something we are doing,
and have done something about, to improve and make sure that we
do use the best methodology.
51. What is the expertise at the LSE public
policy department? As far as I remember them, university departments
are small, filled with people with a grievance against each other,
who hate the outside world and would not know how to talk to a
prisoner if they saw one. What is their qualification for talking
about the report on prisoners?
(Sir John Bourn) I am tempted to say that we make
a contribution to LSE ourselves by giving them the opportunity
to work together in this way; and with the public policy group
which is a mixture of political scientists, economists, statisticians,
operational research people, geographers, psychologists, and they
bring in others to them. I think your comments about the social
sciences in general are very well made. Certainly the LSE itself
has set up a very wide range of interdisciplinary activities as
a centre for economic performance, for example, as well as the
public policy group. Maybe social scientists have come to see
that if they are to prosper they have to find better ways of working
52. They are not redoing the research; they
are looking at your reports?
(Sir John Bourn) That is right, yes.
53. I see. Because some of itthey are
marking the reports as if they were marking an undergraduate essayis
condescending stuff. Some of it is quite hard criticism. I think
this is the public private partnerships, the "Airwave"
project, which I thought had been a bit of a disaster, but I do
not know. They are specifying scope as future prospects for VFM
rather than actual performance to date. There was a let out, which
lets them skirt over any hard-edged critical comments about the
procurement. In the one on the prison service, it says, "The
study team seemed too willing to rely on data provided by the
Home Office on the Prison Service . . . and they should have looked
at other sources". This is quite tough stuff.
(Sir John Bourn) And it is quite right it should be
54. But, what happens to it? Is that incorporated
in the report? Do you then go and look at the other sources?
(Sir John Bourn) If the report is completed it is
a comment related to the future work. But, if it comes out in
the hot review, then we can take it into account. It is a good
point to make, and it is one that we have sought to take into
account in future studies.
55. But unless it is either circulated with
the report or incorporated in the report, it is just for internal
staff. These are quite telling criticisms actually, which go to
the nub of the issue. What happened to these two criticisms?
(Sir John Bourn) They were criticisms which were taken
into account in subsequent work. The report had been completed
and published and therefore could not be altered at that stage.
These were good points to make, but it does not dismiss the total
and overall value of the report.
56. I am glad to see the LSE back me up with
one small matter, Sir John. In this prisoner re-offending report,
under "minor points" they said: "The repeated use
of the photograph of the female prison officer talking to the
male prisoner on pages 2, 4, 5, 6, 7, 18, 26 and 36, . . .detracts
from its overall appearance". We have had this discussion
before, have we not?
(Sir John Bourn) Indeed we have.
57. In one sense they are quite hard hitting.
They say on that: "More use should have been made of the
data collected; a more evidence-based approach; a more specific
and independent analysis of recommendations could have been pursued"
and on "Airwave" they said, "There are some data
weaknesses. We are concerned that at times this report is a bit
vague and the study team withholds analysis".
I see that we have only got these two reports
from them. Do they ever criticise you for the fact that you have
to agree these reports with the Departmentsnot your faultand
that may cut a bit of the edge away from them, and that the Department
is constantly crawling over these reports with you, and saying,
"Oh, leave that bit out" and, "That is a bit too
harsh"? That is a difficulty that you labour under, is it
not? And it makes our hearings better because, you know, you cannot
wriggle away from it.
(Sir John Bourn) As long as they understand the system
that operates, that there is agreement as to the facts. Of course,
there is not an agreement as to the conclusions and recommendations,
which are ours to make, and which are not agreed with the Department.
But very often if you agree the facts, it is difficult not to
agree with the conclusions and recommendations, which is why so
often, when you and other members of the Committee of Public Accounts
ask the Accounting Officer if he agrees with the recommendations,
they say, "Yes". I think that LSE, when they first did
it, did comment on the need to agree the facts, but it is not
a point that they dwell on now, because they see and understand
that the value, as far as the Committee of Public Accounts is
concerned, of having a report, where you do not have to spend
the whole time arguing between the C&AG and the Accounting
Officer as to what the facts are. They have signed up to those
and that is the springboard to go forward.
58. Final question from myself, looking at the
BKR Haines Watts report on your VFM audit of your financial audit
support team, "Technical Advice, Training and Development";
that is an important team, which is why it is being examined.
In paragraph 1.9, it says:
"It is important to consider the policy
for staffing the support function. It is usual for individuals
to stay in the team for around two years, normally as part of
their career progression. However, they often retain other responsibilities,
which serve to restrict their support work, and other demands
further reduce the effectiveness of their tenures, with staff
turnover being relatively high" .
And if we look at paragraph 3.16 we find that,
"relatively high" equals virtually 50 per cent in the
year 2001-02, with the range being between seven months and seven
years. Now for a very important group, that sounds a strange amount
of volatility, and also an unnecessary distraction of their attention
because of other responsibilities and other demands on their time.
I regard that as quite an important criticism. It has not received
a great deal of highlight in the report. What are you going to
do about it?
(Sir John Bourn) I accept it. I think it is a useful
criticism and I am glad to have it. It is partly, of course, because
this team has developed quickly. There is also an aspect of it
that the members in the team enjoy the methodological expertise
that goes with it, but they are men and women who are also keen
to get to the front line. Perhaps I could ask Mr Sinclair, who
runs the team, to say something about that?
(Mr Sinclair) Yes, thank you. It is a
combination of experienced staff and more junior staff. At the
core of the team are three Audit Managers who have been in place
for a very long time, and David Heald is very well aware of some
of the quality that is embodied in those people. But it is a unit
where we, quite deliberately, pick some of our brightest and best
to spend a couple of years on the team, because it is a position
which gives them exposure both to professional methodology issues,
to the profession in terms of standards
59. But a couple of years is not, in fairness,
criticised. It is the ancillary activities they have to do as
well, and the fact that in the last year turnover was up to 50
per cent, with some people serving only seven months.
(Mr Sinclair) There was an individual who served seven
months because they were promoted out of the team. We are talking
turnover within the office, we are not talking turnover external
to the office, and it has been our policy to post them there for
a couple of years, so turnover something like 50 per cent would
be the natural consequence of that. As Sir John says, we are now
looking very closely at the balance here, because it has been
a question of balancing up the value of the central exposure,
the very rapid, wide range of experience it gives those individuals,
and the need to maintain continuity within the team. And we have
listened to what the external auditors say and we are looking
at extending the periods of appointment to that team.