Select Committee on Science and Technology Sixth Report


Research Outcome Measures

7.31 Research methods are not the only important factor in designing trials; the outcome measures used to measure the effect of a therapy also require consideration. Outcome measures can include both objective and subjective measures. Objective clinical measures include the quantification of physiological indices such as blood pressure or the size of tumour. Subjective measures include symptom scores, patient satisfaction scales and quality of life measures. Traditionally RCTs have concentrated on objective findings although, more recently, patient-centred outcome measures have also been used and some well-established questionnaires have been developed. Nowadays quality of life measures can include measures of pain, physical activity, sleep, energy, emotional reactions and social isolation.

7.32 The research undertaken should identify a single or, at most, two principal outcome measures, and the trial should achieve the appropriate level of significance, (e.g. p<0.05 or less). Secondary endpoints can be added. Similarly, composite scores can be used as a primary outcome variable - these would incorporate a number of carefully validated measures, which could be derived by using, for example, principal component analysis, e.g. asthma score, which takes account of symptoms, lung function and treatment requirement.

7.33 During our visit to the Marylebone Health Centre (see Appendix 4), Dr David Peters explained why outcome measures are important and how the Health Centre have tackled the problem of finding measures that can be used in everyday practice. CAM practitioners have utilised a range of questionnaires and interviews. However, many of these instruments required much time and thought from the patients, since the Marylebone Health Centre had decided to opt for a short form of questionnaire that measures only subjective endpoints, which they are now piloting. Their experience has shown that research on appropriate outcome measures that can be used in practice-based research is needed, and this is one of the reasons they are piloting their form.

7.34 The move towards incorporating objective and subjective endpoints into a trial should be reassuring to the CAM practitioner who is concerned with making as broad an assessment as possible of the various changes the treatment has brought about. Vincent & Furnham suggest that there is no reason why measures considered to be especially appropriate for a particular CAM therapy, e.g. changes in the emotional state following homeopathic interventions, should not be recorded and correlated with changes in other measures or indices of clinical change. However, they also suggest that these should not be considered as outcome measures until their reliability and validity have been established.

7.35 The more endpoints used in a study, the higher the chance is of a Type I error. A Type I error is where a change is found on a measure even when, in reality, there has been no improvement. A way of avoiding this is to designate one measure as the primary endpoint and, should this improve, changes on other measures can be examined to confirm any effects of the treatment.

Which Research Method to Use and When

7.36 It is clear that there are many methods available for conducting research into healthcare interventions. The RCCM, which has 16 years' experience in trying to develop CAM research and getting its results accepted, said: "...there has been a debate about the question of RCTs and their application to the area of complementary and alternative medicine. We think that this debate is unhelpful because, essentially, we need to begin with what are the questions we want to ask and then design the appropriate trial and use the appropriate methodology" (Q 26). We agree. The debate over which methods are applicable to CAM and which are not is probably unhelpful; this dilemma has consumed much energy and has produced strong divisions of opinion in the CAM and conventional worlds. The more useful question is which method is suitable for answering which problem.

7.37 Which method is most appropriate to use will depend on the level of development within the therapy and on the particular questions being researched. This was articulated particularly well by the RCCM: "…the establishment of evidence-based medicine requires evidence from both quantitative and qualitative methodologies. Again, to reiterate, the method is determined by the research question. We would suggest that a range of methodologies should also be employed. Health service researchers are increasingly using qualitative methods. And methods employed in the social sciences should also be employed in the evaluation of CAM, depending on the research question. So we may ask, for example, what it is about complementary medicine that people feel is of benefit to them? Is it a genuine therapeutic relationship or is it [related to] where the needles are placed in acupuncture? They require a different approach. One requires in-depth, qualitative interviews. The questions of how does it affect a patient's quality of life, and how does the therapy affect a patient's physical condition would require a more quantitative approach, such as assessment by using a disability scale or a health status measure. So that the full range of methodologies ought to be applied, depending on the research question"(Q 135).

7.38 FIM produces a useful table showing which methods are suitable for which situations in the Discussion Document Integrated Healthcare: A Way Forward for the Next Five Years?

7.39 Mr Michael McIntyre, a trustee of FIM, told us that he believed this controversy over what research methodology should be used was part of the reason why so few CAM practitioners attempted rigorous research: "I think from the CAM side one of the reasons why, perhaps, the amount of research and applications is as low as you say, is that there is a general fear that there is going to be a misunderstanding of the paradigm" (Q 91). It is our hope that, as more CAM practitioners are trained in research methods, and are made aware of the different types of research design; and as more conventional scientific investigators become aware of the intricacies of CAM research, this 'general fear' will be overcome.

Expertise on Grant-Awarding Boards

7.40 Given some of the complexities in designing trials for CAM, it has been suggested by several of our witnesses that there is a particular need for members of grant-awarding bodies to understand the specific problems facing CAM research in order to be able to make a fair and well-informed judgement on the importance and quality of a proposal for funding. The British College of Naturopathy and Osteopathy (P 31) also suggest that to counteract the feeling that many Research Councils do not give CAM proposals a fair chance, funding bodies should recruit CAM members with the appropriate research qualifications to help determine the validity of protocols.

7.41 We asked some of the main funding bodies whether they thought that there was a need for them to appoint specialists in this area on to their boards. The MRC told us: "We do have boards that can judge the proposal. The Health Services Research Board contains a wide range of people, not necessarily practising complementary and alternative medicine but people who understand research methodologies, research questions and can judge whether those questions are answerable using the methodologies that individuals can formulate. I think we do have the people who can judge those things" (Q 1087).

7.42 The Wellcome Trust also defended the composition of their panels: "The panel system that we have within the Trust is a very strong one and is based on peer review. If within the panel there is a lack of expertise we have the option to co-opt an expert to deal with the particular application. If there was a complex CAM application it could then be dealt with by bringing a specialist on to the panel. The panel itself would make the decision, taking account of that expert's advice. The Trust goes to great lengths to get proper peer review. We employ 90 scientific officers to send out all of the bids we receive for peer review. We believe that that is important. To date, we have received 163 CAM applications. The indications are that 37 per cent of those have been funded. That proportion is higher than we would expect for our more orthodox applications, which routinely is about 30 per cent" (Q 1132).

7.43 We asked the Association of Medical Research Charities (AMRC) whether they thought their member charities' boards had the relevant expertise. They too felt that their current provisions were fine and that there was no need to change their system to give CAM proposals a fairer wind: "As an Association we are committed to the use of peer review and believe that that is the best way for charities to make judgements about the best use of their funding. But peer review is quite a flexible system and it should not be applied in a rigid way. The AMRC's guidelines accept that there are certain areas in which one may need a different review process, but the key principle is that there should be an internal and external process of peer review. Where specific expertise is not available on the panel we insist that those charities must seek it externally and choose external referees in an open way. We provide support and advice for charities in identifying external referees. One possibility is for AMRC to draw up a list of potential CAM external referees, although charities do not indicate that they have any difficulty in identifying referees through the normal process of literature researches, networks and various other ways. We also advise charities to go overseas so that questions about the status of organisations and the networks in which particular individuals feature are diminished" (Q 1190).

7.44 The AMRC went on to describe one particular initiative by one of their member charities to aid CAM applications. "The Arthritis Research Campaign is about to introduce a mentoring process of peer review for CAM applications. Even if they are of lower standard initially, applications will be picked up by a member of the panel and taken through with guidance by specialists to try to raise the standard of specific applications. Only very large charities with significant staff can take on that mentoring role. It is an example of how the peer review process can be used to give feedback and to raise the standard of an application so that it can come back again. I do not believe that there is anything inherently wrong with the peer review system for CAM research. I hear criticisms of peer review from every speciality" (Q 1190).

7.45 Overall it would seem that the majority of funding bodies are now willing to ensure that CAM research proposals are reviewed by well-informed individuals. To achieve equity with more conventional proposals, we recommend that research funding agencies should build up a database of appropriately trained individuals who understand CAM practice. The research funding agencies could then use these individuals as members of selection panels and committees or as external referees as appropriate.

Environment and Infrastructure for Research

7.46 There are currently a variety of different environments in which CAM research is conducted in the United Kingdom. These vary from university-based research departments which operate as part of well established medical school research departments, to projects based within charities and in clinical practice either in hospitals or in primary care.

7.47 During the course of our Inquiry we visited three different research environments. Two of these were university-based research departments. The first was the Department of Complementary Medicine at the University of Exeter, which is based within a school of postgraduate medicine and supports the United Kingdom's only CAM Chair. The second was based within a school of medicine — the Complementary Medicine Research Unit of the School of Medicine at the University of Southampton. The third research environment we visited was an NHS clinical practice — the Marylebone Health Centre, an NHS inner-London GP practice which offers CAM therapies alongside conventional care and which supports practice-based research. (See Appendices 3, 4 and 5.)

7.48 We heard much evidence in favour of establishing and supporting a few centres of excellence in CAM research, such as those at Exeter and Southampton, as opposed to spreading funds and resources across many disparate projects. FIM's discussion document Integrated Healthcare: A Way Forward for the Next Five Years? suggests that "it would seem appropriate to concentrate resources on establishing a number of research centres linked with higher education institutes with the capacity to conduct high-quality research into CAM".

7.49 This is the approach of NCCAM in the USA. Dr Stephen Straus, the Director of NCCAM, told us: "The eleven centres we fund to date are really intended, in part, to draw those CAM practitioners and experts into the fold of a larger research enterprise within an established community. Out of the many hundreds of institutions in the United States we are creating foci within only one dozen or so and we hope that we will see leaders in the coming years" (Q 1734).

7.50 The Wellcome Trust also supported the idea of centres of excellence and suggested these should be encouraged to develop from existing centres of research excellence to avoid the delays in generating high-quality CAM research. Dr Howard Scarffe of the Wellcome Trust told us: "I had an opportunity to visit one of the clinical research facilities that we fund at a large university teaching hospital. I was excited that another venture was to be undertaken by the Trust with Government under the Joint Infrastructure Fund. Within 25 yards of that clinical research facility the Wellcome Trust is to fund research laboratories…It is very exciting that attached to a large university teaching hospital campus is a clinical research facility in which all researchers can work together. Adjacent to that is a purpose-built world-class laboratory. We are also funding a director of the clinical research facility so that he or she can give full attention to getting it off the ground. It struck me…that if we had good facilities and researchers we could begin to graft on other bits, of which complementary medicine might be one. If one began to build capacity from the ground level there would be a lag of between 10 and 15 years to train people up to a high level. Therefore, there is a need for a system whereby the research can be grafted on to what is already there and use made of the present expertise" (Q 1141).

7.51 Although concentrating funding in a few centres of excellence has many advantages this does not mean there is no place for smaller practice-based research projects. As previously noted, there are many different ways that CAM research can be conducted and large-scale RCTs are probably best conducted in centres of excellence; qualitative research may be ideal for practice-based research.

7.52 At the Marylebone Health Centre (see Appendix 4) we heard from Dr Sue Morrison, one of the senior partners. Dr Morrison explained that as a practice they favoured rigorous clinical audit and have used such data to develop a manual of integrated care for other practices to use. However, she described some limitations to their data. For instance, some patients self-select the Marylebone Health Centre in order to have access to CAM and therefore wider information is needed from across the Primary Care Group on what patients want from their healthcare and, within this, the role of CAM.

7.53 Dr David Peters at the Marylebone Health Centre described how research has the capacity to serve both practitioners' and patients' needs. For example, audit ensures quality assurance, research through qualitative methods increases understanding of the patients' experience, action research promotes service and professional development and case studies illustrate best practice models. In this way practice-based research promotes quality and understanding.

7.54 We received written evidence from the University of Westminster Centre for Community Care and Primary Health (CCCPH) (P 234) which is also run by Dr David Peters. As well as awarding degrees in various CAM therapies and conducting research in this area, the Centre runs a clinic. They explained the advantages of an educational and research department having links straight into a clinic: "The Polyclinic is creating unparalleled educational and research opportunities where students in the BSc and Masters' programmes will gain practical clinical and research experience under the supervision of some of the United Kingdom's most experienced and best-qualified practitioners…As a multi-disciplinary complementary therapy teaching, research and service delivery resource, the Polyclinic will be unique in Europe and the CCCPH are looking to develop national and international education and research partnerships" (P 235).

7.55 As we discussed in Chapter 5, several of the newer universities now offer CAM courses. The aspiration is that these courses will help establish more university-based CAM research. The BMA told us that they expected these newer universities to be "important players in the field of research in much the same way that medical faculties have a role within medical research" (Q 354). They added: "The question is whether those faculties have sufficient experience yet in devising research protocols, and clearly it is important that they work together to share that experience. Again we believe that organisations such as the Medical Research Council should also be able to offer their help and support in these early stages in the devising of trials and protocols" (Q 354).

7.56 We also asked the CVCP whether they thought the newer universities had the infrastructure to support good quality research. They told us they believed that they did: "The quality assurance regime which the universities operate through the Quality Assurance Agency, which is a tough, self-regulatory regime, would expect every university to consider those issues in respect of any programme: it would expect to ascertain that each university has the appropriate infrastructure which, in certain types of programme, would have to include a research base for mounting a programme. The inspections which are done by the Quality Assurance Agency would certainly cover those areas" (Q 281). They also told us that the regulatory mechanism is there to ensure that no university is left thinking that they will supported if they are mounting programmes without the necessary infrastructure (Q 281).

7.57 From the evidence we have received it is clear that there has been a change of attitude of a few higher education institutions towards CAM as a legitimate subject for both quantitative and qualitative research. However, the small base and fragmentation from which this research will have to be conducted would seem to be a major barrier to progress. We recommend that universities and other higher education institutions provide the basis for a more robust research infrastructure in which CAM and conventional research and practice can take place side-by-side and can benefit from interaction and greater mutual understanding. A preferred model would involve centres of excellence committed to establishing a wider framework of conventional scientists, social scientists and CAM practitioners. These would provide a basis for enhancing research into CAM while ensuring it was of high quality, addressed relevant questions and was integrated with conventional methodology. Advantages would be gained by facilitating multi-disciplinary research with access to medical, psychological, social-scientific and pharmaceutical clinical trials. We recommend that a small number of such centres of excellence, in or linked to medical schools, be established with the support of research funding agencies including the Research Councils, the Department of Health, Higher Education Funding Councils and the charitable sector.

Research Education

7.58 An interest in science or clinical research is not at present a requirement for all CAM training courses, although some schools are introducing courses on research and research projects. Our evidence has helped us construct a picture of the attitude of CAM practitioners towards research. There is an increasing number of CAM practitioners who believe research is important and are willing to put their time and effort into it, but very few appear to have sufficient knowledge or skills in research to advance their interest.

7.59 In the USA one of the main ways that NCCAM is hoping to improve the capacity to conduct CAM research is to improve the education in research methodology of those involved in CAM. This is done through the funding of career development awards at various levels. Dr Stephen Straus, the Director of NCCAM told us: "I believe that those awards need to be to individuals who will be mentored by outstanding scientists and have a protracted period of tutelage, a minimum of three years and ideally five years, to cultivate their skills as an independent investigator" (Q 1734).

7.60 We talked to various United Kingdom funding bodies about the prospect of them awarding research fellowships directed towards students with an interest in CAM in order to invest in the long-term future of CAM research development.

7.61 Although the Government were eager to highlight the importance of education and training in research, they do not currently have a route for supporting research fellowships in this specific area. When we asked Professor Sir John Pattison, Director of NHS Research & Development, how he saw such fellowships being supported, he told us: "I think we have a track record of building capacity in areas of orthodox medicine. Primary care is the first example we embarked upon. In collaboration with the Chief Medical Officer we are about to embark upon similar research training fellowships and career scientist awards in public health…If one simply took those as models for how one would start to build capacity in any area of research to health and health services, then that would be the way to do it" (Q 1869).

7.62 Dr Howard Scarffe of the Wellcome Trust told us he saw the Trust as having a role in supporting programmes which provide training and support in research methodology in the area of CAM. However, the Wellcome Trust's current policy towards awarding PhD studentships is likely to disadvantage CAM applications as applicants without conventional medical or scientific training are likely to be excluded from applying. Dr Scarffe told us: "At present our policy is oriented towards people with medical or scientific qualifications" (Q 1161).

7.63 The Department of Health told us that they saw encouraging research training as one of the best ways of improving the research capacity within CAM: "The approach of the R&D programmes to this and, indeed, all of medicine has been very much to encourage access to training as well as access to research funding. That is provided through a number of mechanisms, including training in research design, training and methodology regionally and centrally. There is a groundswell of movement towards a higher standard of education for researchers. I see that move on the part of the professionals as key to increasing the volume and quality of research that is done" (Q 15).

7.64 The MRC also support research training. They told us: "…our fellowship schemes include fellowships in subjects allied to medicine, and people in complementary medicine are open to apply for fellowships — as they are in…Health Services Research" (Q 1087).

7.65 As well as attracting mainstream experts to investigate CAM, the MRC also explained they have mechanisms for advising people from all areas who need help in designing trials: "...the new MRC Unit on clinical trials, which has a division without portfolio - which is already giving advice to people in trials in areas where traditionally we have been rather weak. That includes advice to particular individuals on how to conduct a trial on complementary medicine" (Q 1085).

7.66 Many CAM university degree courses now include research modules and this is likely to catalyse a change in CAM practitioners' attitudes to research and the need for evidence-based practice. FIM summed up this change: "…we have a lot of university degree courses and very good levels of training being brought into being…and…in nearly every case there is a research module. It does require undergraduates of CAM to…undertake at least some training in research so that the culture of research is encouraged…and that is a very important feature of the education of CAM practitioners — so that we have a common language between researchers in the orthodox field and those in the CAM field" (Q 91).

7.67 Thus, while there currently exist some research training opportunities available to CAM practitioners, none of these is specifically directed towards CAM and very few, if any, of these fellowships have been taken by individuals conversant with the practice of CAM. There may be two principal reasons for this. Firstly, if a practitioner has received no basic education in research methodology he or she is unlikely to seek specialised research training; this is why we hope that CAM regulatory bodies will include research methods in their core curricula (as discussed in Chapter 6) and why the new university courses in CAM represent a promising development. Secondly, the reason why few CAM people take up research training opportunities may well be that they do not know about them. Bodies such as the Departments of Health, the Research Councils and the Wellcome Trust should help to promote a research culture in CAM by ensuring that the CAM sector is aware of the training opportunities they offer. The Department of Health should exercise a co-ordinating role. Limited funds should be specifically aimed at training CAM practitioners in research methods. As many CAM practitioners work in the private sector and cannot afford to train in research, we recommend that a number of university-based academic posts, offering time for research and teaching, should be established.

previous page contents next page

House of Lords home page Parliament home page House of Commons home page search page enquiries index

© Parliamentary copyright 2000