I woke up this morning to news on BBC Radio 4′s Today programme that a report by the Public Accounts Committee of the UK Parliament had found that only half of clinical trials are disclosed.
But the Today broadcast was worse than simply rehashing that old statistic on clinical trials disclosure, as it got various facts wrong which could easily confuse a casual listener. So I’d like to clear up a few things that weren’t clear from Today.
Probably the biggest mistake Today made was to confuse “disclosed to regulators” with “disclosed in the public domain”. According to Today, half of all trials are withheld from regulators. That’s simply not true. Pharmaceutical companies are obliged to make the results of all their trials available to regulators. It’s a condition of getting a drug license.
I think what they meant to say on Today was that half of trials are withheld from the general public. That’s also not true, but it is closer to being true than claiming that they are withheld from regulators.
Here is what happens. If a drug is submitted to regulators for licensing, the regulators get to see all the trials. In an ideal world, all those trials would also be published in the public domain, although in practice, not all of them are. So the failure of disclosure is entirely about what enters the public domain, and not what regulators see. Clinical trials data not entering the public domain is of course a problem, but it’s not the same problem as what was claimed on Today, who alleged that regulators don’t see the data. That would be a far worse problem if it existed, but thankfully, it doesn’t.
So how much data is “withheld” from the public domain?
First, I have a problem with the word “withheld” in this context. It implies some kind of grand conspiracy. In practice, there are many reasons why the results of trials don’t get published. This may be because they are submitted to journals, but then rejected. It may be that they are not submitted in the first place because researchers know that they are likely to be rejected. Either way, there is no evidence that there is a systematic attempt to hide data.
But looking at the actual figures for how much data doesn’t make it into the public domain, the figure of 50% appears to have been accepted by the Public Accounts Committee, as we can read in paragraph 3 of their report. It looks like this was based on evidence from Ben Goldacre, who has been cheerfully promoting this figure as part of the marketing strategy for his latest book. The source he cites for the figure appears to be a report from the NIHR Health Technology Assessment programme published in 2010. There are 2 problems with that. First, the report doesn’t actually say that only half of clinical trials have been published. Nowhere does it report a summary statistic for that measure. Second, although the report was published fairly recently, it is a review of the literature rather than new research, and many of the papers it reviews were published some time in the past. This is important, because publication practices have changed much in recent years, and results are now far more likely to be disclosed than they were in the 1990s when many of the studies in that report were done.
More recent studies tend to show much higher rates of disclosure. This study from 2010 found that 80% of studies were disclosed in the public domain, this study from 2013 (which included studies on some drugs that are not on the market, and so are arguably less of a priority for disclosure) found a 77% disclosure rate, and another study from 2013 (which included only licensed drugs) found an 89% disclosure rate.
The last study was mentioned on the Today programme, in an interview with Richard Bacon MP, one of the members of the Public Accounts Committee. Astonishingly, he claimed that it referred to events that have not yet happened. Here is what he said:
“The 90% figure that the industry’s coming out with is talking about a proposal which was due to come into force in January 2014 which has now been delayed which would make sure that clinical trial data would be released from now on”
Whether he simply was unaware of what the study actually showed and unintentionally got his facts wrong or whether he was deliberately lying is something I wouldn’t like to speculate about. Either way, listeners to the programme would have been badly misled.
Of course, it is true that less than 100% of clinical trials are disclosed, and that is something that needs to be addressed. We should not be happy until the clinical trials disclosure rate reaches 100%. It is also probably true that the recent figures of 80-90% disclosure were not true in the past: failure to publish has become less acceptable in recent years than it used to be in the 1990s. Those historical unpublished trials are also a problem. In an ideal world, they would all be published (though there are of course costs of doing so, and whether the costs outweigh the benefits for some of the older studies is a question that needs to be asked).
Let’s be clear. Disclosure of clinical trials data is important. But so is getting your facts right. I think those who campaign for greater disclosure of clinical trials do themselves a great disservice and substantially weaken their case when they use dodgy data and misinformation to try to make their point.