The strange story of the Tamiflu data

In a recent post about chapter 1 of Ben Goldacre's “Bad Pharma”, I mentioned that the chapter included a strange story about how the Cochrane investigators tried to get access to data on Roche's anti-influenza medication, Tamiflu, which would require a whole blogpost by itself. When I started to look into the Tamiflu story, I realised I was wrong about that. It's actually going to require 2 blogposts to tell the story. This is the first of those posts. In this post, I shall tell the story of the utterly surreal interactions between the Cochrane investigators and Roche. I shall save the question of what the evidence actually tells us about the efficacy of Tamiflu for another day.

The Tamiflu story started a while back, and indeed I blogged about it in 2009. Briefly, the problem arose when the Cochrane Collaboration (a normally splendid organisation that does a great job of looking carefully at the evidence for medical interventions to find out whether they really work) were preparing a systematic review to determine whether Tamiflu was an effective treatment for flu. It turned out that some important clinical trials of Tamiflu had not been published, and the story concerns the Cochrane investigators' attempts to obtain the data from Roche.

As I wrote in my previous blog, it seems to me that both the Cochrane investigators and Roche behaved very badly. That situation has continued, and I have to say it fills me with despair that both sides have taken the thoroughly unreasonable positions that both of them have. In fact it probably fills me with even more despair that I even need to write the phrase “both sides”: surely scientists should all be on the same side? Science is not a football match.

At one stage, Roche offered to make the reports available if the Cochrane group would sign a confidentiality agreement. They refused. I don't see the need for a confidentiality agreement, so I think Roche behaved badly in demanding one, but I equally don't see the justification for refusing. If the Cochrane investigators had been interested in scientific truth, rather than scoring points over the other side, then I can't see why they couldn't have just analysed the data under conditions of confidentiality.

Some commentators, such as Goldacre, have defended the Cochrane reviewers' refusal to sign the confidentiality agreement. On page 83 of Bad Pharma, Goldacre writes of the confidentiality agreement: “This was an impossibility for any serious scientist”. I agree it's not desirable, but impossible? Surely not. In fact the Cochrane handbook acknowledges that there are times when it will be necessary to sign confidentiality agreements before receiving data, so it seems bizarre for Cochrane reviewers to claim that it's impossible.

Anyway, in the end, I'm not sure what ever happened about the confidentiality agreement, but Roche did put the clinical study reports on a website so that they would be available to the Cochrane reviewers. If you're already familiar with the Tamiflu story, this might surprise you. Much of the popular narrative gives out the story that the reports have been kept a closely guarded secret. For example, on page 88 of Bad Pharma, Goldacre says “Roche had promised: 'full study reports will also be made available on a password-protected site...' . This never happened.”

The British Medical Journal have also perpetuated that myth: their website states “In Dec 2009, Roche publicly promised independent scientists access to "full study reports" for selected Tamiflu trials, but to date the company has not made even one full report available.”

This is rather puzzling, because as far as I can tell, Roche have indeed made the reports available. I can think of two possible explanations for the discrepancy. One is that Roche have made it so difficult to access the reports that they might as well have not made them available. If you visit the website where Roche have posted the reports, you are first greeted with a security warning telling you that the site's security certificate has expired. This does not exactly inspire confidence.

If you continue past the security warning, you find that the reports are available, but they are only available if you have a password. You have to agree to various onerous conditions to apply for a password, one of which is that you don't work for a commercial organisation. Even then, that doesn't seem to be enough. One of the people I follow on Twitter, Dr Dave Briggs from the University of Manchester, which I'm fairly sure was not a commercial organisation last time I checked, attempted to get a password to download the reports, but was unable to do so. Roche didn't refuse as such, but the mechanism to get a password simply didn't work, and Roche made no attempt to help him, despite repeated requests.

So frankly, any kudos Roche might have deserved from making the reports available is comprehensively nullified by the thoroughly unhelpful way in which they have done it.

But I don't think the difficult access to the reports is the reason why the claim that the reports are not available is so widely made, because if you read the 2012 Cochrane review carefully, it appears that they did have access to the study reports.

I think the reason why the popular narrative seems discordant with what has actually happened is that those claiming that Roche did not make the full reports available are not being very clear about what they mean by “full reports”. A few words of explanation are probably necessary here for those who are not familiar with how clinical study reports are usually written.

Data from clinical studies may be presented in various forms. They may be published as papers in peer reviewed journals, or they may be presented in clinical study reports. Clinical study reports have a synopsis, which tells you the main things you want to know about a study, but may not necessarily have enough detail to let you do a proper meta-analysis. The synopses for the Tamiflu reports are available online.

The synopses, which are typically a few pages long, are just one small part of a full study report. This would typically extend to hundreds of pages, and would include very detailed descriptions of the methods and results of the study, with many data tables. Those reports would contain far more detailed information than you would ever hope to find in a paper published in a journal, and would almost certainly give you more than enough detail to do any reasonable meta-analysis.

In addition to the main body of the study report, there are also appendices. They contain supplementary information, such as a copy of the protocol, details of the ethics committee that approved the study, certificates of analysis for drug batches, and so on. On the whole, they don't add much to the understanding of the study: they contain information that regulators need so that they can check regulations have been followed. But they are not necessary to be able to interpret the results of the study. The main body of a study report is intended to be readable as a stand-alone document.

So as far as I can tell, the complaint that Roche have not made “full” study reports available arises because Roche have made available only the main body of the reports, and not the appendices. To my mind, claiming the full reports are not available is a little misleading, as the main body of the report really should tell you everything you need to know, unless you are are regulator.

In any case, given that a clinical study report contains vastly more data than a published paper, and that most systematic reviews are done perfectly happily using nothing more than published papers, I find it utterly baffling to find it suggested that the Cochrane reviewers could possibly be missing any of the information that they need.

One of the Cochrane reviewers, Tom Jefferson, did attempt to explain this on Ben Goldacre's blog in a discussion of Tamiflu data, saying “Missing from the site are the remaining modules including protocol, certificates of analyses, SAP and other documents vital for the understanding of each trial.” However, he did not explain why they were vital for the understanding of each trial, despite my asking him that specific question. I don't think it would be too hard to explain what specific piece of information was missing, if any important information was missing, would it? A protocol is of course an important document, but any half-way competently written clinical study report would reproduce all relevant information from the protocol, so it should not be necessary to read the protocol as well to understand how the trial was conducted.

In fact I asked the same question on a blog from another of the Cochrane reviewers, Carl Heneghan, but he also failed to answer.

So we are left knowing that the Cochrane investigators believe that important information is missing (which seems unlikely if they have access to the main body of the study reports), but they have never specified what information is missing. Do they not know the dosing schedule? Do they not know how the randomisation list was generated? Do they not know how many patients were recruited but then excluded from the analysis? All those things would typically be included in the main body of a clinical study report, but if some of them are missing, it should not be too hard to specify which.

So what does the latest Cochrane review itself tell us?

Sadly, it doesn't shed much light on this strange state of affairs. The authors describe (see page 11 of the pdf version of the review) a rather complex process for assessing the completeness of information. It's quite confusingly written, but as far as I can tell, the authors required not only that all study reports needed to have complete information on a number of items, but that it also had to be possible to verify the information in the clinical study report from an external source (eg a regulatory summary report). If that was not possible, then the authors simply excluded the study from their analysis.

To my knowledge, that is unique in any systematic review.

It's not clear where this method came from. I have not been able to find it described in the Cochrane handbook as a recommended method. Given that most Cochrane reviews are based purely on published papers, to require a complete regulatory report together with confirmation of the information from external sources before studies can be included in a review would probably mean that there would not be a single Cochrane review including any studies at all. It seems a ridiculously high bar to set. It would be interesting to know if the authors of this Cochrane review are of the opinion that every other Cochrane review that has ever been produced is untrustworthy. That would be the only logical conclusion of their method.

The authors of the Cochrane review do not report which part of this process led to exclusions of papers. They merely tell us that there was insufficient information to include the studies. So we are still none the wiser about what information is missing.

So let's have a brief recap of what's going on here.

Much fuss has been made, by folk such as the British Medical Journal, Ben Goldacre, and the Cochrane Collaboration, about lack of access to Tamiflu data. It is strange to single out Tamiflu. Lack of access to data is a problem in clinical research in general, as I have blogged about more than once before.

But the data on Tamiflu are far more available than on many drugs. While it's disappointing that many of the Tamiflu trials remain unpublished in peer-reviewed journals, that situation is certainly not unique to Tamiflu. It's important to note that all Roche's trials on Tamiflu have been made available on their website in summary form to anyone, and their clinical study reports have also been made available to the Cochrane investigators. There are probably not many drugs which have been disclosed to systematic reviewers to a greater extent than Tamiflu has.

So it's all very strange that the Cochrane investigators are claiming that they do not have sufficient access to Tamiflu data and that Tamiflu is being used as a poster child for lack of transparency from the pharmaceutical industry.

While any attempt by me to explain the reasons why they are doing that would be speculation, it is interesting to note one or two things in passing.

The lead investigator of the Cochrane review, Tom Jefferson, has been found collaborating with anti-vaccine conspiracy theorists in a rather disturbing way. Is it possible that there is some desire to denigrate the pharmaceutical industry, rather than to engage in a scholarly examination of the evidence? I don't think an anti-industry bias within the Cochrane Collaboration can be ruled out, given that another recent Cochrane review drew some very strange anti-industry conclusions that seemed to have little relationship to the results they found.

And what of others who have picked up the story? Well, Ben Goldacre has a book to sell. Telling stories of how the evil pharmaceutical industry hide all their data is probably good for business. The BMJ also tend to take a rather anti-industry stance on many things (although they deny this). Is the Tamiflu story just a convenient peg on which to hang a few general rants about evil pharmaceutical companies?

I honestly don't know what's going on here. I do realise that the previous couple of paragraphs read a bit like some of the conspiracy theory rants you read from anti-vaccinationists and homeopaths, albeit in reverse. Maybe I'm way off base with all of this.

But still, you have to admit that the Tamiflu story is all rather odd, don't you?