Select Committee on Work and Pensions Minutes of Evidence

Examination of Witnesses (Questions 200-219)



  200. Put bluntly, actually the ONE pilot approach is not hitting the objective which was set out for it in the beginning of this dual approach?
  (Professor Marsh) This was the evidence that the clients gave us, and people's memories are not infallible—this was some five months after the interview so people have struggled sometimes to remember what was discussed—and since many of them were still on benefits perhaps benefits was in the front of their minds. But it was not memorable. This was on average a 32-minute interview, it would have to be a pretty memorable 32 minutes for them to say, "Yes, they talked about jobs and this was the job ..."

  201. I see that point, but of course in the research you are doing about the interview, you can tell fairly quickly to what extent the emphasis was on getting into work and to what extent it was about the availability of benefits and so forth.
  (Professor Marsh) It did not succeed in emphasising work as the main thing that was spoken of.

  202. Can I just ask you one other point, your latest published research, Cohort Two, Wave One, suggests there was an employment effect for sick and disabled people but not for other groups?
  (Professor Marsh) Yes.

  203. The In-House Report on the employment effects of ONE, which is called number 88, uses administrative benefit records to conclude there was an employment effect for lone parents but not for other groups. Can you just explain to us, without getting too technical, what conclusions we should draw from the different findings there?
  (Professor Marsh) You should focus on the size of the differences. The In-House Report was using every single flow-through, which would have an end of about 90,000, anything that twitched would probably have been used. I, myself, am not fully qualified to study the macroeconomics that were done there, I do surveys. In this very large sample a closed sample of lone parents moving through the system were interviewed at five months, and there was no difference at all between the groups being piloted and controlled, but there was in the first stage, in the voluntary stage. We found a difference for the sick and disabled at the compulsory stage, which was quite noticeable. I think for the same reasons, if I may say so, novelty, that the lone parents were introduced to ONE in the voluntary stage at exactly the same time Working Families' Tax Credit was introduced, and they picked up on it quicker in the ONE areas. In the second interview the good news had spread to the control areas too, here is a new novelty. This is the first time that sick and disabled claims were being required and confronted with an interview, and that may have had a kick-start effect.


  204. Staying for a moment with the question of the whole evaluation process, if I said to you that I have quite a strong memory of a representative from Deloittes, one of the managers there, of one of the pilots, Richard Granger. He actually said that he very much looked forward to seeing properly studied technical data that gave him as a businessman some idea of whether there had been success in the objectives in the pilot areas. I am getting from you the sense that there are an awful lot of health warnings with the results that you are bringing to us this afternoon. That is not a criticism of you, because you said yourself to do this properly you would have started earlier and done it in other ways. What value can we actually attach as a Committee to the professional work that you have done and what are the limitations? Is Mr Granger likely to be disappointed as a businessman trying to say, does this add up as a sensible reform for Government to make? You, presumably, have no input at all into any cost benefit questions, for example? You would not be able to measure value for money, if we said it cost £68 million between 1999 and 2001, social policy researchers would not have a view about that either way?
  (Professor Marsh) That research is being separately done and it will incorporate some of our research in it.

  205. I want you both to tell me to what extent the hard value of the research that you have done is safe for us to rely on?
  (Professor Marsh) Let me be very firm about this, what this evaluation lacks in design—necessarily it did not have an element of random assignment or any of those new tricks people can do—it makes up for with resources. It is an extraordinarily well resourced piece of research. We have the numbers which enable you to understand the process and we have the numbers to be able to apply post hoc statistical controls. It just makes it very, very hard work to do all of the extremely complicated statistical manipulations which we have to do to control out the likely other sources of variation, comparing pilots and controls, when you do not have observations before the experiment.

  206. Well resourced, quantitative studies I can understand, what about qualitative? Did you find that you could make up for some of the design faults, if you like, by the fact that you had enough depth to do the qualitative stuff?
  (Ms Johnson) To reinforce what Vicky Davies said earlier, from our work it was always intended that by undertaking in depth interviews with a sample of clients, it was approximately 100 interviews at each round, that our findings would be used alongside the quantitative findings to help explain why things happened the way they did for which groups of clients. That is the way that you need to treat our findings, that you can look to us with questions about why things happened the way they did, for which clients, which goes very much hand in hand with the quantitative data.

  207. What is next? What else can we expect by way of research? Do you know of other pieces of research that you or other people are doing in the same area that we might have some access to?
  (Ms Davies) We are currently doing fourth round research of work with ONE clients, particularly the impact of annual triggers on lone parents, but also taking a more detailed look at some of the different groups within the sick and disabled category to find out what distinctions can be drawn from there. That also includes some follow-up interviews with clients that we spoke to in the last year and a half, so it will provide you with long-term impact to see what progression has happened over time, the longitudinal impact.

  208. To be published when?
  (Ms Davies) April or May.
  (Professor Marsh) We are currently analysing Cohort Two, Wave Two, and that will be available shortly. For lone parents there will be Cohort Two, Wave Three and they will be contacted again and new data gathered on them. After 18 months we really will have a very clear portrait of what has happened to them.

Miss Begg

  209. Just very briefly as a follow-up to the qualitative research, do you get down to the fine detail of where the personal adviser came from, whether he came from the Employment Service or the Benefits Agency, and whether there is a difference in the quality of their advice and whether that is an explanation as to why in some cases there is job focus and in other cases it is benefits and whether it comes from the quality of the training of the personal adviser?
  (Ms Davies) We do not get that information at all because we only speak to clients, we do not have access to the staff side at all.

  210. Would that not be very useful for the Department to know, because that obviously must influence the kind of training that they will need when the thing is rolled out?
  (Ms Davies) Yes, it would. Other research has being going on and Tavistock may be presenting it to you later.
  (Ms Johnson) We were not able to systematically know where the personal adviser came from, but at the same time we said there were cases when we were talking to clients that they reported that the PA had said to them that they had previously worked in the BA and in the ES and because of that they were or were not in a good position to provide particular sets of advice about benefits or moving towards work. It is an issue you need to take account of.

Mr Stewart

  211. In its memorandum to this Committee the Department of Work and Pensions commented on research and said, "Clients had a strong expectation that ONE would mean that their benefit claim would be dealt with more speedily, but this was not generally their experience". Can you tell us what your research found about benefit processing?
  (Ms Davies) Our research itself did not address benefit processing. We were talking to clients themselves and we were reliant on them expressing their individual experiences. We did not talk to the BA staff or to the ONE staff that would be responsible for processing claims at all. We do not have the broad statistics which show whether the length of the process has been reduced as a result of ONE. The main thing we can say is that where there were delays in receiving benefits clients expressed considerable frustration, often because they did not know when their money was coming. Again, it comes back to the financial security issue, that often prompted clients to go back to their personal adviser to find out what was going on or, alternatively, going to specialists that deal with either income support or incapacity benefit to get further information. As a whole, that is not something that we can comment on in great depth.

  212. Professor Marsh, is that the same for you?
  (Professor Marsh) It took longer, 3.4 weeks compared with 2.9.

  213. One of the objectives of ONE was to improve the assessment and delivery of benefits by providing a more individual service to claimants tailored to their needs. What does your research suggest about the quality of information and the advice that claimants received and the quality of services regarding benefits more generally?
  (Professor Marsh) The valuation is higher in the pilot areas compared with controls. In the pilot areas people generally found the advice they got more helpful, more likely to be new advice, rather than stuff they had heard before, advice that left them feeling more hopeful and advice that treated them as an individual, not as a number. Generally speaking they felt they had been treated very well. Some of those differences are quite small. On that last one I mentioned, for example, among lone parents 35 per cent thought they had been treated very well compared with 21 per cent in the conventional system of control.

  214. There is quite a considerable difference there between the ONE and the non-ONE pilot?
  (Professor Marsh) More had a meeting, so the aggregate has increased.

Ms Buck

  215. Just going back to the function of the personal adviser, it has come over strongly, certainly from your research, that people were expecting the benefits element to be the key role of the adviser, where did they get their impression from generally?
  (Ms Davies) It was not so much necessarily getting the impression from somewhere, it was more what was their primary concern. Why were they making the claim, and the information they may have been told, did not necessarily sink in.

  216. They did not go into a session with a personal adviser thinking this is going to be different particularly?
  (Ms Davies) No.

  217. Okay. Did they feel they had enough time?
  (Ms Davies) With their personal adviser? It varies again in individual cases. There is also a sense that clients often felt rushed or that their personal adviser was incredibly busy and so was not able to spend longer with them to discuss more detailed enquiries and to explore work in more detail, again it is mixed.
  (Professor Marsh) At least half thought they had got all of the information they needed.

  218. About benefits?
  (Professor Marsh) Yes. About anything.

  219. What has come out from what you have been saying is that the experience is particularly focussed upon benefits. What I am trying to get at is whether the research tells us anything about people expecting the process to be about benefits and work and the extent to which the interview, by not addressing the work component as much, was driven by the client's expectation and the adviser's expertise or just sheer lack of pressure and time?
  (Professor Marsh) In these flow samples— it is not often one gets such a large flow sample so you can tease out the process— a lot are new clients. They have come to claim benefit; it is an important event in their lives; it is the first time they have done it in many cases; what else would they expect to talk about? They have their start-up meeting. They are told you have to come back in three days and have this meeting or you will not get any benefits. They assume they are going to talk about their claim.

previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2002
Prepared 26 February 2002