Clinical trial disclosure: myths and realities
I woke up this morning to news on BBC Radio 4's Today programme that a report by the Public Accounts Committee of the UK Parliament had found that only half of clinical trials are disclosed.
That's a statistic we've heard before. And as I've explained more than once before, it's not true. Nonetheless, zombie statistics are very hard to kill.
But the Today broadcast was worse than simply rehashing that old statistic on clinical trials disclosure, as it got various facts wrong which could easily confuse a casual listener. So I'd like to clear up a few things that weren't clear from Today.
Probably the biggest mistake Today made was to confuse "disclosed to regulators" with "disclosed in the public domain". According to Today, half of all trials are withheld from regulators. That's simply not true. Pharmaceutical companies are obliged to make the results of all their trials available to regulators. It's a condition of getting a drug license.
I think what they meant to say on Today was that half of trials are withheld from the general public. That's also not true, but it is closer to being true than claiming that they are withheld from regulators.
Here is what happens. If a drug is submitted to regulators for licensing, the regulators get to see all the trials. In an ideal world, all those trials would also be published in the public domain, although in practice, not all of them are. So the failure of disclosure is entirely about what enters the public domain, and not what regulators see. Clinical trials data not entering the public domain is of course a problem, but it's not the same problem as what was claimed on Today, who alleged that regulators don't see the data. That would be a far worse problem if it existed, but thankfully, it doesn't.
So how much data is "withheld" from the public domain?
First, I have a problem with the word "withheld" in this context. It implies some kind of grand conspiracy. In practice, there are many reasons why the results of trials don't get published. This may be because they are submitted to journals, but then rejected. It may be that they are not submitted in the first place because researchers know that they are likely to be rejected. Either way, there is no evidence that there is a systematic attempt to hide data.
But looking at the actual figures for how much data doesn't make it into the public domain, the figure of 50% appears to have been accepted by the Public Accounts Committee, as we can read in paragraph 3 of their report. It looks like this was based on evidence from Ben Goldacre, who has been cheerfully promoting this figure as part of the marketing strategy for his latest book. The source he cites for the figure appears to be a report from the NIHR Health Technology Assessment programme published in 2010. There are 2 problems with that. First, the report doesn't actually say that only half of clinical trials have been published. Nowhere does it report a summary statistic for that measure. Second, although the report was published fairly recently, it is a review of the literature rather than new research, and many of the papers it reviews were published some time in the past. This is important, because publication practices have changed much in recent years, and results are now far more likely to be disclosed than they were in the 1990s when many of the studies in that report were done.
More recent studies tend to show much higher rates of disclosure. This study from 2010 found that 80% of studies were disclosed in the public domain, this study from 2013 (which included studies on some drugs that are not on the market, and so are arguably less of a priority for disclosure) found a 77% disclosure rate, and another study from 2013 (which included only licensed drugs) found an 89% disclosure rate.
The last study was mentioned on the Today programme, in an interview with Richard Bacon MP, one of the members of the Public Accounts Committee. Astonishingly, he claimed that it referred to events that have not yet happened. Here is what he said:
"The 90% figure that the industry's coming out with is talking about a proposal which was due to come into force in January 2014 which has now been delayed which would make sure that clinical trial data would be released from now on"
Whether he simply was unaware of what the study actually showed and unintentionally got his facts wrong or whether he was deliberately lying is something I wouldn't like to speculate about. Either way, listeners to the programme would have been badly misled.
Of course, it is true that less than 100% of clinical trials are disclosed, and that is something that needs to be addressed. We should not be happy until the clinical trials disclosure rate reaches 100%. It is also probably true that the recent figures of 80-90% disclosure were not true in the past: failure to publish has become less acceptable in recent years than it used to be in the 1990s. Those historical unpublished trials are also a problem. In an ideal world, they would all be published (though there are of course costs of doing so, and whether the costs outweigh the benefits for some of the older studies is a question that needs to be asked).
Let's be clear. Disclosure of clinical trials data is important. But so is getting your facts right. I think those who campaign for greater disclosure of clinical trials do themselves a great disservice and substantially weaken their case when they use dodgy data and misinformation to try to make their point.
It is indeed important that someone dispassionate has a balanced view of the totality of data about a new drug before it is released and that is indeed the case now and has been for some time.
Those campaigning for full release of all studies in the public domain are after something else, perhaps they want to second guess the regulators, perhaps they just don't like the horrid capitalist drug companies, perhaps there is another reason.
Full release at the level of detail seen by the regulators would make it very hard to guarantee patient confidentiality and then people would not want to participate in trials.
I'm sure you are aware of the statistical methods for recreating missing studies and testing if released studies are representative.
There is one other wrinkle here David. Large double blind studies are an expensive 'gold standard' for drug trials. But are they a gold standard? Increasingly I minded to think that they are not the future. The confounding factor, a huge one, is that we (being rather difficult) have very individual responses to the more complex pharmacological interventions. I seem to recall that the prime architect of the double blind recanted on his deathbed much as Sigmund Freud did! Unfortunately I do not remember his name. Over to you?
George. Sorry to have taken so long to reply.
I agree with your point about the 'value' of the randomised double blind study. While those studies do have their merits they ignore the benefits of adaptive studies and they have been so overwhelmed by regulators that they have become impossibly expensive.
It is now recognised by many that a completely new approach to drug development and approval is required if new molecules for personalised medicines to address complex system failure diseases are to be developed at reasonable cost and quickly. The present system is no longer affordable or productive enough.
I don't see however that full release of all data is part of the solution.
[...] Adam’s links on zombie stats [...]
Some of the most prominent researchers on publication bias this week published a paper on missing trial results in The Lancet, part of the Lancet's landmark series on waste in medical research.
You can read the paper here, it includes the lead author of the NIHR review on publication bias.
As you can see, they are very clear in their summary that the best estimate is that half of studies have gone unreported.
http://press.thelancet.com/research4.pdf
http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)62296-5/abstract
It is very peculiar - and rather cheap - to see Adam Jacobs so energetically insisting that this figure is somehow part of the marketing campaign for a book published over a year ago. I've spent an enormous amount of time over the past year building the AllTrials.net campaign. Nobody pays any of us to do that, and it's been extremely heartening to see the campaign bearing fruit, you can follow the progress at badscience.net if you're interested.
Adam Jacobs' claim that the trials completing very recently are less likely to go unreported is misleading in two ways. Firstly, he is cherry picking individual studies, and there is no review article that gives a clear overview of the evidence, or looks at time trends. Secondly - as has been repeatedly explained to Adam - the vast overwhelming majority of prescriptions in clinical practice today are for treatments that came on the market over the past five, ten, fifteen, twenty years, not the past three. This is why the AllTrials campaign is absolutely clear about asking for retrospective transparency, getting complete access to trial results from 2014 will do nothing to improve the evidence base for everyday clinical decisions in medicine for a very, very long time.
Any increase in the proportion of trials that have been reported over the last couple of years would be welcome, if and when there is clear evidence of that, without cherry picking. But it is miserly progress, and it will - like the currently proposed initiatives for full transparency from 2014 onwards - represent minimal impact on clinical practice today.
Again, that is why the AllTrials is concerned about the rate of missing trials for the research that informs current clinical practice, and that is why we call for retrospective transparency. It seems that is easy for everyone but Adam Jacobs to understand, and it is beginning to seem like he is more interested in muddying the waters than joining the broad coalition moving towards a solution. If I sound intemperate or rushed, it is only because this has all been explained to Adam very many times, and it is now hard to continue assuming good faith.
Thanks for stopping by, Ben.
"Some of the most prominent researchers on publication bias this week published a paper on missing trial results in The Lancet, part of the Lancet’s landmark series on waste in medical research.
"You can read the paper here, it includes the lead author of the NIHR review on publication bias."
I take if you're familiar with the "appeal to authority" fallacy? If you want to argue about the numbers, then argue about the numbers themselves, not about who said what.
"As you can see, they are very clear in their summary that the best estimate is that half of studies have gone unreported."
Yes, they do say that. And they are wrong, for the reasons I explain in my post. They have ignored the two 2013 studies.
"it’s been extremely heartening to see the campaign bearing fruit"
How do you know the campaign is bearing fruit? Sure, we are moving in the direction of transparency, but we were doing that anyway. How do you know AllTrials is responsible for that?
I take it you're familar with the "post hoc ergo propter hoc" fallacy?
"Adam Jacobs’ claim that the trials completing very recently are less likely to go unreported is misleading in two ways. Firstly, he is cherry picking individual studies"
Which studies have I missed out? If you can point me to some good quality recent studies showing much lower rates of disclosure, then I will happily admit that I was wrong, but I'm not aware of any.
"Secondly – as has been repeatedly explained to Adam – the vast overwhelming majority of prescriptions in clinical practice today are for treatments that came on the market over the past five, ten, fifteen, twenty years, not the past three."
And I have never tried to deny that. Please read my blogpost again and look at the penultimate paragraph.
I take it you're familiar with the "straw man" fallacy?
"Any increase in the proportion of trials that have been reported over the last couple of years would be welcome, if and when there is clear evidence of that, without cherry picking."
There is clear evidence of that. If I've cherry-picked the evidence, then tell me what evidence I've missed.
"Again, that is why the AllTrials is concerned about the rate of missing trials for the research that informs current clinical practice, and that is why we call for retrospective transparency."
A call I have consistently supported, so again, you are resorting to a straw man argument to say I don't understand it.
"it is beginning to seem like he is more interested in muddying the waters"
No, I am simply interested in using accurate statistics. I really find it odd that I need to explain that to you of all people, Ben.
Oh, and while you're here, Ben, and we're talking about straw men, you accused me on Twitter the other day of using a straw man argument, but you didn't say how I'd misrepresented you. Could you elaborate?
My position isn’t complicated, and I think it’s shared by the others involved in AllTrials, the authors of the Lancet paper, etc: missing results have a significant impact on public health; this must be fixed; the best currently available evidence estimates that around half the trials on the treatments we use today have gone unpublished. You have invented some kind of different position, something about newly approved medicines that are barely used, and then ascribed it to people including me (and claimed that your invented position is part of some kind of sales campaign for a book from 2012, which is rather cheap).
You have been working in industry for, I believe, an entire generation longer than I’ve been around. It’s a great shame you didn’t take time out of your dayjob in the 1990s, or 2000s, to join others and start your own successful campaign building a consensus on publication bias, engaging with legislators, industry, the media and patient groups to try and fix this problem, outside of your paid activities. I don’t mean that snidely: it’s possible you would have done a great job, better than us, and the problem might have moved forwards much more two decades ago. But you didn’t, so I guess we’ll never know. It’s beginning to look a little as if you’re spending a lot of time trying to derail and slow down a successful campaign on this issue with fractal sillliness around a position you yourself have invented.
So can we be clear about what your position is, Ben?
I think we're all agreed that it was at least approximately true that in the past, only half of trials were published. I have no argument with you there.
What is in dispute is whether that figure has changed in recent years. I claim that it has, and that trials completed in the last few years are much more likely to be published than trials from the 1990s.
It seems to me that you dispute that claim, and you maintain that the chance of a clinical trial being published remains around 50%. So if I got that wrong, now is your chance to clarify.
Do you agree that trials completed in recent years are considerably more likely to be published?
Funny you should mention taking time out of my day job in the 1990s or 2000s to start a campaign. I did exactly that, trying to take action against ghostwriting in medical journals. Granted, that's a different problem to incomplete publications, but I hope you'll agree it's also an important problem, and it's an area where I thought there was a realistic chance that I could actually make a difference. And indeed it seems that ghostwriting is becoming less common, though of course it's hard to ascribe causality based on an n=1 uncontrolled experiment.
Since you are apparently completely unaware of my work on trying to banish medical ghostwriting to the dustbin of history, here are a couple of publications that came out of it that you might be interested in.
As above, I don't think I've expressed a view on whether the publication rate is improving for the most recent trials, that are less relevant to the everyday treatment decisions on drugs today. Although you keep pretending I have.
You haven't expressed a view on whether the publication rate is improving for the most recent trials?
So what did you mean on page 54 of Bad Pharma when you talk about "fake fixes" and said "the problem of missing data has not gone away"? That sounds to me like expressing an opinion on the most recent trials.
And what of this comment on my blog last month, where you cite a 2013 study? That also sounds to me like it's expressing an opinion on the most recent trials.
And then your recent Guardian article said this:
"A 2010 review article by the NHS's own research body summarises the results of a dozen more studies on the same subject: this found that, overall, the chances of a completed trial being published are roughly 50%"
The use of present tense there would certainly suggest to the reader that you believe the chances of a completed trial being published are currently 50%. And then later in the same paragraph, you say:
Large studies from the past two years, chasing up results from huge registries of completed trials, report similar results.
Again, that looks awfully like expressing an opinion on the recent situation to me.
And I believe you put your name to the AllTrials briefing document, did you not?
What did you mean by this statement on page 2:
"However, evidence collected since the NHS NIHR HTA review shows that the problem persists at very high rates"
if you weren't expressing a view on current publication rates?
I do find it odd that you're now claiming you have expressed no view on whether the publication rate is improving. If that's really true, then I have to say that a lot of the stuff you've written has been rather badly expressed.
Hi there. As I've said, I’m not inimical to the idea that publication rates are increasing, I’d be delighted, but you could find a couple of studies showing that rates of missing trials are low at any time in the past, so I’d wait for a good systematic review that looks at time trends in all the evidence. However, if it were true that a quarter of trials went unpublished, for the newest and therefore the least used medicines, that’s still huge, and it wouldn’t change anything if the figure for missing trials had turned out to be a quarter in the past either (as in the Lancet paper, and elsewhere, it's a half). This would still be a very significant public health problem that urgently needs to be addressed, that has been neglected, and that many are expending a great deal of effort trying to kick into the long grass. I think it’s very sad, and peculiar, that you are so energetic about trying to derail the work that’s being done here, apparently acting in bad faith with repeated silly comments about selling books when everyone is working on AllTrials outside their paid work, and tenuously ascribing positions to people that they don’t hold. There is a great deal that could have been achieved in the time that you are taking people away from productive work with this, with your silly claims that evidence wasn’t withheld on Tamiflu, and so on. I really don’t know any more whether that is your intention, which saddens me, because I’ve enjoyed our chats in the past. Best wishes.
"I’m not inimical to the idea that publication rates are increasing"
I'm very pleased to hear you say that, Ben. In that case, I think we are in agreement. As I have said many times, I do completely agree with you that we need 100% disclosure. 77% disclosure, or even 97% disclosure, isn't enough.