Examination of Witnesses (Questions 1
WEDNESDAY 8 MARCH 2000
1. Today we are taking evidence on the Comptroller
and Auditor General's Report examining how the Ministry of Defence
accept equipment off-contract and into service. We have four witnesses:
Kevin Tebbit, Permanent Under-Secretary of State; Sir Robert Walmsley,
Chief of Defence Procurement; Mr John Oughton, Deputy Chief of
Defence Logistics; and Vice Admiral Sir Jeremy Blackham, Deputy
Chief of the Defence Staff (Equipment Capability). Nice to see
so many of you here, gentlemen. I hope it is because you take
it seriously and not simply you trying to out number us! Because
there are long and involved issues in a number of these matters,
we will have our normal session and at the end of it there will
be a closed sessionand I expect there will be from what
I have seen so farwithout the public here. Mr Tebbit, let's
start in the conventional way as you are used to. Paragraph 2.5
records that in half of the cases examined by the National Audit
Office where the equipment failed to meet the Staff Requirement
your Department made concessions which removed from the contractor
the obligation to make good the shortfall. Paragraph 4.10 states
that, given the contractor will usually price his work at the
outset against the contractual requirement, it is surprising that
you have not pursued financial recompense for making concessions
in more cases. Why have you not pursued financial recompense more
(Mr Tebbit) Can I first say the reason
I have come with such large numbers is because my colleagues represent
the new way of dealing with acquisitions and procurement of equipment.
This is, as it were, the Smart procurement head team although
I as Accounting Officer clearly accept full responsibility.
2. I was simply teasing you.
(Mr Tebbit) I thought I ought to explain that. I think
the story is indeed, as you say, one where in some areas there
were financial shortfalls, but I think it is only fair to put
it in the context of the total study. The NAO looked at some £10.7
billion worth of equipment over a five-year period and of that
the financial shortfall, or the area where we could have sought
recompense and did not, amounts only to £60 million worth
of that equipment. Two-thirds of that was in fact one system,
the 966 radar,
so against that background the issues and the areas where we did
not seek direct financial compensation are, I think, very much
the exceptions rather than the rule. There will be a variety of
reasons for doing so. In some areas the contractors met the requirement
and it was an obligation the Department accepted to conduct any
further changes as were required, but there is no general pattern
there. As I say, the pattern is of satisfactory acquisition rather
than failures. I would be happy to discuss or with my colleagues
to discuss any particular financial shortfall areas you are concerned
3. I am sure other members of the Committee
will want to pursue that matter. I will press on as I have a lot
to cover today and I may come back at the end or in the closed
session. I note from paragraph 3.5 that over half of the projects
surveyed by the National Audit Office had difficulty in demonstrating
aspects of equipment performance at the time of accepting off-contract.
The same paragraph records that most projects then undertook additional
trials before the equipment could be accepted into service and
that in 40 per cent of these trials further problems emerged when
comparing performance with the Staff Requirement. Why do you find
it so difficult to design acceptance strategies which address
both the needs of users and contractual concerns?
(Mr Tebbit) These are, by and large, extremely complex
systems that are being introduced into service and some of the
equipments are by their very nature impossible to verify until
one is trialling them in an initial acceptance period. Naval systems,
for example, ships, have to be brought on in that way. You may
also be pointing out I will not say weaker areas, but in the period
which the NAO had covered, we did indeed have two rather separate
processes, one acceptance off-contract, the other one acceptance
into service. The one coming from, as it were, the manufacturer
acquired by the Defence Procurement Executive, the other the frontline
saying it met their needs or not. And it is that dichotomy or
rather the relationship between the two that is at the heart of
Smart procurement, Smart acceptance. That is one of the problems
that we are indeed overcoming into the way in which we are developing
equipment now. So it may be to some extent that the difficulties
were because there were two different audiences within the Department
involved. In the future there is only one audience in an integrated
form with Admiral Blackham here as a central customer for equipment
operating in conjunction with the Defence Procurement Agency and,
indeed, with the in-service support logistics structure which
Mr Oughton represents in an integrated project form, so we look
at all of these things from start to finish looking at performance
throughout the project rather than straightforward requirement
4. I hear what you say, Mr Tebbit. I must say
both from what is in this Report and from what we have seen in
previous Reports and indeed anecdotal evidence, the tolerance
of under-performance strikes me, frankly, as a culture of low
standards, not necessarily one that is just extant today but over
a long time in the MoD. I do not want you to give me an answer
to that at the moment but I would like you when answering other
questions that I am sure will come up on that to explain how you
are going to change what strikes me as a cultural low standard.
(Mr Tebbit) I would be happy to do so.
5. Let me move on. Paragraph 3.26 states that
where some form of remedy or concession was needed, the Department's
analysis of acceptance options was largely qualitative and consensus
reached through discussion and argument between the main stakeholders.
Given the problems that can occur where subjective judgments on
acceptance have been incorrect, as happened with the radio interference
problem on Sea King Mark3a helicopter detailed in paragraph 3.29,
why do you not undertake more quantitative analysis in assessing
whether to accept equipment off-contract and into service?
(Mr Tebbit) A quick word from me and then I might
ask for Robert Walmsley to comment in a bit more detail.
6. Absolutely. I was waiting for your striker
to come into play!
(Mr Tebbit) It is the case now under our new arrangements
that every aspect of procurement and acceptance will have built
into it a plan as to how we are going to verify the equipment
at all stages, not just initial entry into service but also through
life maintainability. We will be operating from what we call integrated
test evaluation and acceptance plans. So the point has been and
is being addressed, as the NAO recognise. It was never the case
that we only had qualitative judgments earlier and now quantitative
ones, but there is a shift in the quantitative direction. Perhaps
I can ask Sir Robert Walmsley to comment on the more specific
(Sir Robert Walmsley) The first thing to say, Chairman,
is that we are extremely careful to quantify any suggestion that
the shortfall threatens safety. That is meticulously documented.
I think you mentioned the Sea King radios and the Military Aircraft
Release for those aircraft very carefully documents what those
radio shortfalls are. What I think the Report makes clear, because
it says it, is that these were "irritating effects".
They are quite difficult to fix. Anybody who has driven a car
and passes a transmitting station like this will have experienced
exactly the same effect. It is very difficult to fix close to
a high-powered transmitter. We do think we have got the solution.
It is under way and we expect to be able to demonstrate satisfactory
performance in due course. As to documentation, it is fully documented.
7. Others will come back to this and I will
certainly come back to one of the points you made in a moment.
In taking evidence on the 1997 Major Projects ReportI think
you were with us Sir Robertwe heard about the technical
difficulties which you had in procuring TUL/TUM Land Rovers which
pulled to the left on braking. I remember you brought us the rubber
bush that caused the problem. Paragraph 3 of Box 4 details the
problems which you had in the earlier procurement of another utility
truckthe RB44which again revolved around the problems
with braking. In the RB44 case you spent almost £1 million
modifying the braking system prior to acceptance. How is it that
the braking problem re-occurred once the vehicle was in service
and why do you seem to be unable to prevent such basic technical
difficulties re-occurring? My own handwritten note here says it
is not rocket science. Mr Tebbit?
(Mr Tebbit) You have got one here which I agree is
partly our fault but I will ask Sir Robert Walmsley to cover it
8. There we are, a hot pass, Sir Robert!
(Sir Robert Walmsley) It is not rocket science, Chairman,
but it is an extremely complex interaction between suspension
and the front axle and the effects of braking. I cannot explain
why it is so hard to make trucks, whether medium or heavy, that
brake in a straight line. As to your specific question as to why
did we keep on thinking we had fixed it and then find we had not,
I think there is a clearer story. First of all, after the initial
production vehicles came off the line we tested them. We asked
for it to be fixed. Because we had asked for a specific problem
to be fixed we introduced extra tests and it was those extra tests
which showed that there was still a problem. We then fixed that
and put the vehicles into service; that was the first stage. Additional
tests uncovered additional faults. The second stage was that in
operation we discovered that the brakes required far too frequent
adjustment in order to keep them up to scratch in terms of performance
and it was not really a practical proposition for the Army to
keep on doing this continual adjustment, so in some vehicles where
they did not continually adjust the brakes the performance was
not adequate, that is why we reverted to automatic break adjustment
which had been in the original specification but which we had
removed. The third problem is that a lot of these vehicles were
kept in store. As part of the defence cost studies we decided
it was not sensible to go on maintaining everything we kept in
store as though it was in operational use. What we missedI
cannot explain itis that some seals on the brake cylinders,
which anyone who has tried to fix their brakes will know, perish
if they are kept in store. If we had not conducted the maintenance
these vehicles would be brought back into service and the braking
problems exhibited. So they are sensitive to not following maintenance.
We did not and that is why there was a problem. We follow the
maintenance now. The vehicles now brake properly.
9. I hear what you say. One has to say, however,
that interactions between mechanical systems are the sort of problems
that designers in the commercial vehicle industry solve all the
time. As for storing vehicles, one often sees lots of vehicles
stored in the open and they are sold afterwards and they are not
expected to go wrong. I think we are going to find people coming
back to this question because it seemed to me to be quite straightforward,
as we discussed last time you were with us.
(Sir Robert Walmsley) And they were commercial vehicle
manufacturers, Chairman. It was them who did not understand the
effect of the bushes and it was them who did not understand they
needed a tie rod between the axle and the structure of the vehicle.
10. It was your Department who set up the contract
with them and you should have demanded that from them.
(Sir Robert Walmsley) We do not specify how to build
vehicles, Chairman, we specify the performance we want and the
performance was found to be inadequate.
11. I am sure others will want to come back
to that one in some detail. Paragraphs 3-6 of Box 5 on page 36
explain the problems you had validating the performance of the
software in the FADEC system on the Chinook MR2 helicopter. In
most cases the people who understand an item of software the best
are those who designed it. Even employing other firms of consultants
will not necessarily give you the knowledge and understanding
you require. Given the shortcomings in the processes followed
on the Chinook MR2 programme highlighted in the box and the fact
that most defence equipments are now more or less software dependent,
what approach does the Department adopt to gain sufficient assurance
that the software in equipments which you are accepting off-contract
and into service will perform as required?
(Mr Tebbit) We use our own people within the integrated
project team. When a project is developed there is a specific
emphasis on any software aspect of it, whether in embedded chips
or with a more general IT application. That is taken as a discrete
element in the project and is followed in detail by the team itself
or through contracts to DERA, the Defence Research Establishment
which we often use for these things, or external consultants,
or industry itself all performing to standards of safety which
are nationally and internationally recognised to make absolutely
certain that what we bring into service actually works and to
make absolutely certain that what we specify can be tested and
proven. There is a whole industry which ensures that this happens,
12. Let us read paragraph 4 of Box 5 on page
36 where it says: "On the basis of EDS-SCICON's work"this
is on behalf of Boscombe Down"in October 1993, Boscombe
Down advised the Department that they could not recommend Controller
Aircraft Release for the Chinook Mk2 because of the `unquantifiable
risk associated with the unverifiable nature of the FADEC software,'
and concluded that rewriting the software was essential."
Why was it not rewritten?
(Mr Tebbit) It was not rewritten because the design
authority for the system informed Boscombe Down that the technique
that Boscombe Down was seeking to use, which is called static
code analysis, was inappropriate for verifying or validating the
software provided because it had been designed to a different
philosophy and testing programme using dynamic testing methods.
In other words, whereas the static code analysis that Boscombe
had chosen to use was looking at the actual design, the Design
Authority made it clear after this that they were testing the
safety performance of the design and did so through 70,000 hours
of dynamic testing. It is important to be clear about what this
meant. Boscombe Down and SCICON and the Department did not say
that the FADEC software itself was unsafe. What Boscombe Down
said was that they were impressed by the handling and performance
of the aircraft using this method. Their concern, as I say, was
that they were unable to verify it using their preferred methods
of testing, but it was made clear to the Department by the Design
Authority that this was not the right method to be using and subsequently
this was demonstrated partly by ourselves using a more dynamic
method for verification which was indeed done, partly by the evidence
of the US themselves in running a very similar system with the
US Forces which had about 95 per cent compatibility of software
which had been introduced in 1990 and partly the evidence that
the system was recommended to be and subsequently proved to be
ten times more reliable and efficient than the previous method
of controlling these engines which was by hydrodynamic methods.
13. What is that based on, that ten times more
(Mr Tebbit) That was the difference between the predicted
improvement in reliability on the previous engines that Chinooks
used in the way of managing those engines.
14. Can you let us have the figures that you
are basing that on?
(Mr Tebbit) That is very simple, Mr Chairman.
15. Do you know what a rare subroutine problem
(Mr Tebbit) I am sorry?
16. Do you know what a rare subroutine problem
(Mr Tebbit) I cannot say I do.
17. It is what happens when you have two subroutines
in a piece of software which are both called "rarely".
As a result, when you are testing that software they do not show
up very often. If they interact badly when they do show up it
can lead to an unpredicted error. As I understand it the static
code analysis was a logical analysis to see whether there was
a potential clash between parts of the software. The dynamic analysis
is effectively a test programme. I read this and I asked the Director
of EDS-SCICON to tell us what this meant to them. Let me just
read you one component of a note sent to us. He said: "EDS
would not have expected to find any Category 1 anomalies in the
software they were testing and would have expected far less Category
2 anomalies than were listed. He noted that EDS-SCICON had been
sufficiently concerned about the safety implications of their
findings to stop testing and to raise their concerns immediately
with Hawker Siddeley (who produced the software under sub-contract
from Textron-Lycoming) rather than waiting to produce a final
report having tested all the software. Mr Bullock told us that
EDS-SCICON were unhappy with the response they received from Hawker
Siddeley who dismissed the concerns as being trivial."
Are you still happy with what you said to me about the safety
of the software?
(Mr Tebbit) I repeat, Mr Chairman, neither
EDS-SCICON or Boscombe Down found the software to be unsafe. What
they said was that they could not verify using their preferred
methodology. The Ministry of Defence then had to consider what
to do about it. What it did was go back to the design authority
and ask for their comments. Their comments were that this was
a different philosophy and a technique which would not work with
this system in the way intended to demonstrate safety. They assured
us that their system was safe and there was the proven record
with the US Forces already. The Ministry of Defence decided to
proceed in a way which imposed a weight limitation on the Chinook
just in case there was any possibility that there could be any
difficulties with the engines. So they proceeded on the basis
of a weight limitation of 18,000 kilograms which meant that even
if one of the two engines failed it would still be perfectly safe
to fly. As it happened none of those occurrences occurred and
it was unnecessary to have done that. Nevertheless that limitation
remained in place, partially at least, until four and a half years
later when it was finally lifted in full. There have been no problems
18. How many aircraft of the US fleet have this
brand of FADEC?
(Mr Tebbit) I would not know the answer about the
19. You were citing it as a reason to take this
as a safe system.
(Mr Tebbit) As I said before, the US fleet has got
a system which is 95 per cent the same. There has not been one
problem with the UK fleet and it has now been flown for something
like 119,000 hours.
3 Note by Witness: Of the £60 million
financial shortfall, we did not pursue financial compensation
for three-quarters (not two-thirds) and that was in respect of
one system, the Radar Type 996, not the Radar Type 966. Back
Note by Witness: It is in fact, The Defence Evaluation
and Research Agency. Back
Note: See Evidence, Appendix 1, pages 31-42. Back
Note: See Evidence page 5; Appendix 5, page 64; and Appendix
1, pages 31-42. Back