Public Accounts CommitteeWritten evidence from the Cochrane Neuraminidase Review Group

We are the team responsible for the NIHR-funded Cochrane review on Tamiflu, referred to extensively in the committee hearing. We would like to take the opportunity to set the record straight, with regard to inaccurate statements that arose in the oral evidence taken before the Public Accounts Committee on 17 June 2013.

Point 1

Dame Sally Davies reported: “Having read it, I do not believe—Kent and Andrew may have other views—that the Cochrane review on Tamiflu is the last answer, not just because they do not have all the data, but, first of all, because they made up their hypotheses once they had got data, and that is not standard research practice.”

Point 1 Response:

With regard to the statement, “making up their hypotheses,” our review followed the standard Cochrane methodology of first writing a protocol of how we plan to conduct our analysis. This protocol was then peer-reviewed, revised and published in the Cochrane Library in January 2011. However in the process of conducting the review we found multiple discrepancies and other unexpected observations in the clinical study reports and regulatory documents as compared to the published data alone. Based on these unexpected findings we developed “post-protocol hypotheses” which we labelled as such to be transparent about the timing. The results from testing these hypotheses are clearly labelled as post-hoc in the review and conclusions are appropriately tentative.

In terms of standard research practice, it is wholly appropriate to perform such analyses, and the methods of these analyses were subjected to peer review before publication.

It is important to recognize that such analyses do not affect the primary and secondary outcomes, but they attempt to understand whether the estimates of effect sizes are modified by biases in the trial design.

For example, the placebo capsules in the Tamiflu trials contained dehydrocholic acid, a compound which is not present in the active treatment, and is known to cause diarrhea if taken at high doses. We therefore hypothesized that it was important to determine if the presence of dehydrocholic acid increases the incidence of gastrointestinal harms in the placebo arm, thus reducing the apparent differences between the two arms in the trial. We therefore formulated a post-protocol hypothesis concerning this to test out the possible effect.

A similar example relates to the content of GSK’s placebo for the trials of their antinfluenza drug zanamivir (Relenza). While reviewing the US FDA critique of zanamivir, we noted their concern over the apparent drop in lung function following zanamivir inhalation, which causes bronchospasm in susceptible individuals and was contained in both the active and the placebo blisters. This principle of using a matching placebo is of course correct, but may have had the effect of increasing the incidence of bronchospasm (or asthma-related episodes) in both arms. This is clearly reported as a warning in the 1999 FDA-approved labelling: “Because the placebo consisted of inhaled lactose powder, which is also the vehicle for the active drug, some adverse events occurring at similar frequencies in different treatment groups could be related to lactose vehicle inhalation”. It was logical therefore to test whether certain harms might be related to placebo content.

A further protocol analysis was developed because in the course of reading the Roche Tamiflu clinical study reports, we learned that some of the trial populations deemed influenza infected was determined after randomization into the study, based on the results of laboratory testing by culture and/or antibody rise, rather than at baseline as we had believed. We noted that in all trials the proportion of patients deemed infected with influenza was lower in the active arm compared to the control arm. Our analysis indicates that this is a systematic problem probably due to oseltamivir suppressing the antibody response to influenza.

It is vital that placebo and active groups of patients have the same chance of being classified as influenza infected, otherwise any comparison between groups deemed to be infected with influenza will be potentially affected by bias and will essentially be a non-randomised comparison.

This is an important issue, particularly if the trial medication affects the production of antibodies; the selection of the influenza infected population is confounded by taking the trial medication. All efficacy analyses in the original trials were conducted in the influenza infected population, and almost all of Roche’s published journal articles of the treatment of influenza report on this subpopulation of individuals deemed influenza infected without any explanation of the differences in numbers between the two arms of the studies.

Point 2

Dame Sally Davies: “They [our Cochrane group] extracted data from 25 studies but excluded 42 and took no data from published studies.”
Chair: “They took no data from published studies?”
Dame Sally Davies: “No, and I could go on.”

Point 2 Response:

This is an important point, as it could be misunderstood to mean that we either purposefully or inadvertently excluded important trials in the public domain from our review. This is not the case.

It is important to understand that not all Roche Tamiflu trials have been published in the scientific literature, and not all trials that have been published have been published accurately. However for all Roche Tamiflu trials, very detailed reports called “Clinical Study Reports” do exist and are typically thousands of pages long each. In the course of updating our review over the past years we identified and reported important discrepancies between the way a particular trial was reported in published literature [reference 1,2,3,4] versus how it was reported in the far more detailed clinical study reports, our Cochrane review therefore decided to use the clinical study reports as the primary document to extract the data from, to ensure the most complete and accurate analysis possible.

For example: the two most cited published trials of oseltamivir either did not mention serious adverse events (Nicholson 2000, published in The Lancet), or stated that “... there were no drug related serious adverse events” (Treanor 2000, published in the Journal of the American Medical Association, JAMA).

Indeed, these findings were repeated by bodies such as the UK NHS: “No serious adverse events were noted in the major trials and no significant changes were noted in laboratory parameters” (UKMIPG 2001).

However, the equivalent clinical study reports for these two trials (known by their study identifiers WV15670 and WV15671, respectively) describe 10 serious adverse events (in nine participants), some of which were classified as “possibly” related to Tamiflu.

Enquiries with the first authors were unrewarding as no one had apparently seen the raw data and at least one report had been ghost written (Cohen D. Complications: tracking down the data on oseltamivir. BMJ 2009;339:b5387).

Based on these experiences, and Roche’s promise to release full clinical study reports for at least 10 of its trials as well as new freedom of information policies at the European Medicines Agency, we made a decision to base our review on clinical study reports and not journal publications or conference abstracts. For example, we decided not to use the only published material for trial M76001, which is a conference abstract of around 300 words in length, despite this being the largest treatment trial ever undertaken on Tamiflu (with just over 1,400 people of all ages). We note that this is the only document published in the public domain for this trial.

In terms of this 300-word abstract, we do not know who actually wrote it. This preclude its use over that of the clinical study report.

As part of its investigation into Tamiflu decision making, Channel 4 News found that Professor Treanor—the only author named on the conference abstract—said that he didn’t actually participate in study M76001 and doesn’t remember presenting it at a meeting in 2000. Channel 4 put these facts to Roche and Dr David Reddy, Roche’s Global Pandemic Taskforce leader, responded: “It’s not infrequent that you may have somebody who authors but they don’t actually present it at a conference, it depends upon their availability.” (D Cohen, BMJ 2009)

Therefore, given these issues with the published data, and given our finding that 60% (3145/5267) of patient data from randomised, placebo controlled phase III treatment trials of oseltamivir have never been published, it is wholly appropriate for us to refer to and use the clinical study reports as the principle source document.

For study M76001 this decision meant that we used the 1,514 page clinical study report that we obtained from the European Medicines Agency via freedom of information requests as opposed to data from a 300 word published abstract. Just this month, we received the remaining portions of the M76001 clinical study report, and altogether it is 9,825 pages long.

Therefore, the statement that we included 25 studies but excluded 42 is incorrect. In the published Cochrane review we state:

“For 42 studies we were unable to obtain sufficient information to determine their suitability for further assessment and analysis in our review (see Characteristics of studies awaiting classification). Rather than exclude these studies outright we have decided to retain them pending confirmation of data from the additional clinical study report modules. For the oseltamivir trials (WV15799; WV16193; WV15759/WV15871 (WV15799; WV16193; WV15759/WV15871; WV15819/WV15876/WV15978; MV21737; JV15824; NV16871; MV22841; WV15825; MV21118; JV15823; WV16277; ML20589) we wrote to the manufacturers seeking validation of aspects of methods and results of the trials but received no answer. According to our rules these trials had not been validated and we excluded them from entering Stage 2 of the review.”

Of note, 20 oseltamivir trials were not available for data extraction. However in April 2013, Roche informed the Cochrane Group that it would release redacted clinical study reports for 74 trials of Tamiflu that it sponsored. We are now in the process of receiving these reports.

We find it perplexing that the regulators continue to state they had all the available evidence. To the best of our knowledge, we had all of the oseltamivir trial data the European Medicines Agency had received, which we obtained via a freedom of information request (22,239 pages of data) (see Table 1).

Table 1

THE PROPORTION OF CLINICAL STUDY REPORTS WE OBTAINED FROM THE EUROPEAN MEDICINES AGENCY

Trial	No. participants		Parts (“modules”) of oseltamivir clinical study reports obtained (dark circles) and still not obtained (light circles)	Study pages obtained
JV15823	316		Synopsis only	32
JV15824	308		Synopsis only	19
M76001	1,459		?????	1,514
ML16369	478		No data obtained	0
ML20542	534		No data obtained	0
MV21879	862		No data obtained	0
NP15757	59		?????	445
NV16871	329		“Core Report” and “Study Documentation”1	614
NV25118	9		No data obtained	0
WP16263	400		????	8,545
WV15670	726		?????	1,032
WV15671	629		?????	1,018
WV15673/15697	1,562		?????	804
WV15707	27		????	458
WV15708	385		????	661
WV15730	60		????	525
WV15758	698		?????	1,126
WV15759/15871	335		?????	1,121
WV15799	962		?????	900
WV15812/15872	404		?????	683
WV15825	572		?????	875
WV15876/15819/15978	741		?????	973
WV16193	808		????	894
WV16277	451		No data obtained	0
Total	13,114			22,2392

Note that these 22,239 pages largely comprise incomplete clinical study reports (only one study report was complete), but this is because the European Medicines Agency itself only had incomplete reports so could not provide compete reports. We are therefore unsure who besides Roche had the complete study reports that we are currently receiving thanks to Roche’s new promise, and where these data were held at the time of regulatory approval.

Other groups, independent of ours, have noted the weakness in the available evidence, particularly with regard to the claim made by Roche that its clinical trials show that Tamiflu lowers the risk of complications. Burch et al (2009), in their UK funded HTA project “Overall, little information was available on the effects of either drug on the incidence of complications, and there were very few events, in both the healthy adult and at-risk populations. Furthermore, weaknesses in the available evidence limit the reliability and the ability to generalise any results relating to the effect of these drugs on the rates of complications.” (Burch J, Corbett M, Stock C, Nicholson K, Elliot AJ, Duffy S, et al. Prescription of anti-influenza drugs for healthy adults: a systematic review and meta-analysis. Lancet Infect Dis 2009;9:537–45.)

We would like to conclude that in our review, even with such a huge amount of data available, we acknowledge that we still do not have and cannot convey the full picture.

However, the Cochrane team undertaking this work has completed in excess of 100 reviews across multiple areas of health care, and is involving extensive expertise into this review process. The review, to date, has the most extensive data set ever used in this area and as such provides a transparent assessment and outlines the important threats to validity of the trial results.

We therefore stand by our current conclusions: “We found a high risk of publication and reporting biases in the trial programme of oseltamivir. Sub-population analyses of the influenza infected population in the oseltamivir trial programme are not possible because the two arms are non-comparable due to oseltamivir’s apparent interference with antibody production. The evidence supports a direct oseltamivir mechanism of action on symptoms but we are unable to draw conclusions about its effect on complications or transmission.”

We are currently receiving full clinical study reports containing study protocol, statistical analysis plan and individual patient data to clarify outstanding issues.

We would therefore welcome, if called upon by the Committee, the opportunity to set the record straight.

Signed by Cochrane Neuraminidase Review Group

Carl Heneghan
University of Oxford, Department of Primary Care Health Sciences, Oxford, Oxfordshire, UK
Correspondence to carl.heneghan@phc.ox.ac.uk

Tom Jefferson
The Cochrane Collaboration, Roma, Italy

Peter Doshi
Johns Hopkins University, Division of General Internal Medicine, Baltimore, Maryland, USA

Mark Jones
Queensland Health, Centre for Healthcare Related Infection Surveillance and Prevention, Brisbane, QLD, Australia

Chris Del Mar
Bond University, Centre for Research in Evidence-Based Practice (CREBP), Gold Coast, Queensland, Australia

Rokuro Hama
Japan Institute of Pharmacovigilance, Osaka, Japan

Matthew Thompson
University of Oxford, Department of Primary Care Health Sciences, Oxford, Oxfordshire, UK

References

1. Jefferson T, Jones M, Doshi P, Thompson M, Henegan C. Ensuring safe and effective drugs: who can do what it takes? BMJ 2011;342:c7258

2. Doshi P, Jones M A, Jefferson T. Rethinking credible evidence synthesis. BMJ 2012;344:d7898 doi: 10.1136/bmj.d7898.

3. Jefferson T, Jones M A, Doshi P, Del Mar C B, Heneghan C J, Hama R, Thompson M J. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database of Systematic Reviews 2012, Issue 1. Art. No.: CD008965. DOI: 10.1002/14651858.CD008965.pub3.

4. Doshi P, Jefferson T, Del Mar C (2012). The Imperative to Share Clinical Study Reports: Recommendations from the Tamiflu Experience. PLoS Med 9(4): e1001201. doi:10.1371/journal.pmed.1001201

20 June 2013