The zombie statistic that just won't die
I have written more than once before about how the commonly heard statistic "50% of all clinical trials are not published" is nonsense. It is a zombie statistic: no matter how many times you try to kill it, it just keeps on coming. And now it turns out its supporters are becoming increasingly dishonest in their attempts to defend it.
The zombie statistic appeared again in the last few days on the Pharmafile website. This was presented as being news, but in fact it described a report that contributed very little new data, and mainly described a publication from last December. The publication, needless to say, did not show that only 50% of clinical trials are published, but if you read it really carelessly, are determined to believe that only 50% of clinical trials are published, and don't care too much about what the study actually showed, then you might interpret it as meaning that.
What I find really disappointing here, however, is not just that the same old flawed statistic is being rehashed, but that Pharmafile seem determined to resist any attempts at debunking. Jackie Marchington wrote a nice explanation of why the "50% of trials are unpublished" statistic was nonsense, and posted it as a comment to the Pharmafile article.
Sadly, the comment didn't last long before it was deleted with no explanation. Jackie posted a further comment to ask what had happened to the first one, which was also deleted.
Deleting comments from websites when they disagree with you is a sure sign of a weak argument and intellectual dishonesty. The "50% of clinical trials are unpublished" brigade are surely losing the argument if deleting critical comments is the best they can do.
If you want to read what Jackie had to say, happily she had the foresight to post her comment here as well.
Just seen Adam's tweet "ICYMI yesterday, the intellectual dishonesty of supporters of "50% of trials are published" statistic bit.ly/1kb4T2A #alltrials"
This is very peculiar behaviour. The best currently available evidence shows that trials on the treatments we use today are routinely and legally withheld, and about half go unpublished. This is based on huge, extremely onerous and labour intensive pieces of research such as this, which chased up results on 10% of all trials registered on clinicaltrials.gov, the biggest registry in the world (and found half missing).
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2728480/
This landmark 2014 review in the Lancet by the leaders in this field of research might be helpful for anyone interested to understand why missing trials are important, I've posted the abstract below in full, there's a free PDF of the full paper if you Google around. (You'll see the leading academics in this field of research are happy with "half" as an estimate.)
http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)62296-5/abstract
"The methods and results of health research are documented in study protocols, full study reports (detailing all analyses), journal reports, and participant-level datasets. However, protocols, full study reports, and participant-level datasets are rarely available, and journal reports are available for only half of all studies and are plagued by selective reporting of methods and results. Furthermore, information provided in study protocols and reports varies in quality and is often incomplete. When full information about studies is inaccessible, billions of dollars in investment are wasted, bias is introduced, and research and care of patients are detrimentally affected. To help to improve this situation at a systemic level, three main actions are warranted. First, academic institutions and funders should reward investigators who fully disseminate their research protocols, reports, and participant-level datasets. Second, standards for the content of protocols and full study reports and for data sharing practices should be rigorously developed and adopted for all types of health research. Finally, journals, funders, sponsors, research ethics committees, regulators, and legislators should endorse and enforce policies supporting study registration and wide availability of journal reports, full study reports, and participant-level datasets."
"Intellectual dishonesty" is a very odd and strong accusation. It's notable that Adam has devoted so much time and effort on trying to derail the public discussion on this issue, with crescendoing fury. This is clearly a high priority for him.
The trials of the past 5, 10 and 20 years are the trials on the medicines we use today. These are the trials we need to make informed decisions about which treatment is best. It is a serious public health problem that these trials have been so casually withheld, and that problem is being maintained by efforts such as Adam's to distract attention and derail.
Thanks for stopping by again, Ben.
I've already explained to you why you're wrong when you insist that about half of trials go unpublished, so rather than responding to your same arguments again, I shall merely refer you to my previous explanation.
So moving on to new ground, I see you don't like my accusing the Pharmafile website of intellectual dishonesty.
So how would you describe it when someone deletes a comment from a website for no apparent reason other than that they disagree with it? Would the intellectually honest way of dealing with it not be either to admit they were wrong or make an argument about why they think they were right?
Deleting comments from websites because you disagree with them is the modus operandi of intellectually bankrupt people such as homeopaths and other quacks. I'm surprised to see you defending that behaviour.
"So moving on to new ground, I see you don’t like my accusing the Pharmafile website of intellectual dishonesty... Etc....etc.. the modus operandi of intellectually bankrupt people such as homeopaths and other quacks. I’m surprised to see you defending that behaviour..."
That's silly, I've no interest in the handling of comments on a pharma industry magazine's website.
Your tweet said a lot more than that.
https://twitter.com/dianthusmed/status/443369306731331585
I think you're over-reaching, in your desperate and energetic campaign to derail the campaign for trials transparency, and now you're casually throwing around grand allegations of dishonesty. Your claims about other peoples claims on trials transparency have been discussed here before: you're inventing detailed views and then attributing them to others, here and elsewhere, as already discussed:
http://dianthus.co.uk/clinical-trial-disclosure-myths-and-realities#comment-124504
It's a very odd way to prioritise your time, and a big waste of everyone else's.
"That’s silly, I’ve no interest in the handling of comments on a pharma industry magazine’s website."
Well that's odd, because that's what this blogpost was about.
As you know, and as is clear from your tweet alleging intellectual dishonesty, this blog post is part of your energetic campaign to muddy the waters on trials transparency. You repeatedly claim the finding that half of trials for the treatments we use is a "zombie statistic", and your evidence for this is that you believe - in the absence of a systematic review - that transparency for trials from the past couple of years may be improving. These trials are almost entirely irrelevant to clinical practice, because the vast overwhelming majority of prescriptions are for medicines that came on the market (and were researched) in the preceeding years. This notion that trials transparency can be fixed by only giving access to new trial results, prospectively, is of course the position of EFPIA, PhRMA, and so on. It's unclear what part of "all" and "trials" you don't understand, but everyone else finds it very easy to follow.
As for your claim that only half of trials are published, Ben, we've been through all this before, and again, I refer you to our previous conversation.
As I'm still not sure you've understood my allegation of intellectual dishonesty, let me make it clear that it is a very specific allegation about the behaviour of the editors of the Pharmafile website. If you want to discuss that, I'll be delighted to hear your views, but you said in a previous comment that you weren't interested in it.
I can assure you that I have no wish to muddy any waters. As you know full well, I am completely in favour of full transparency around clinical trials. But if you and others continue to misuse statistics, then I will continue to challenge you on them.
Hi Adam. Thanks for the post - I'd like to clarify the issues here, as it seems not everyone gets it.
The original issue was that a shoddy Pharmafile article reported on a poor Thomson Reuters Cortellis article. The Pharmafile story was inaccurate. The Thomson article was wrong.
So, what are you supposed to do?
I politely pointed out (well, I think I was polite but no-one will ever know, because Pharmafile deleted my comment) the inaccuracies of the story, and the fact that despite the opening sentence, the Thomson article that had been incorrectly described as a "new study", was, in fact, neither new, nor correct. The Pharmafile story annoyed me on a number of levels. Firstly, "someone on the internet was wrong" (OK, I'm sorry, I can't help it); secondly, the Thomson article was being cited as "new data" that supported the "50% of all trials are unpublished" statement; thirdly, the Thomson article was incorrectly citing an earlier study about a subset of data with CT.gov results postings... why am I doing this again? Read my comment (sorry Adam, I don't know how to embed links in your comments box, so I'll just post this http://zombiestatistics.co.uk/blog1.php/seriously-that-quot-50-of instead).
So I commented on the Pharmafile story (hey, isn't that what comments are for? If you don't want comments then go in to your backoffice admin and disable them) explaining why it was wrong, and pointing out some other inaccuracies. And my comment was deleted, without explanation. And my query as to why my comment had been deleted was also deleted, without explanation.
So this is how zombie statistics are propagated - by a lazy "journalist", parroting an inaccurate article, and who doesn't want to discuss or defend their "work".
The source article by Thomson Reuters Cortellis was a perfect example of how zombie statistics are born. One reference, misrepresented or misinterpreted, then cited as an authoratitive source. At least Thomson have followed up with me (OK, so it was a sales call because I downloaded the article, and when I explained why I downloaded it, they promised to refer the issue to the editorial team [not heard yet, by the way]).
So I suppose my point is, if you must insist that "50% of all trials remain unpublished" (in the interests of accuracy, the headline was actually "Half of all trials registered on a US clinical study database are not being published in public journals") please at least cite data that you consider to be about "all trials".
Quick update - I have been contacted by the editorial team at Thomson Reuters Cortellis and they have corrected their article to accurately reflect the findings of the Riveros article. Big thumbs up to them for wanting to provide accurate information.
Not sure what the Pharmafile guys will do, but I can guess. (...um, nothing?)
It's interesting to see journalists closing ranks. (Maybe it's all a conspiracy of persons named "Ben". This is Ben Adams on Ben Goldacre. http://www.pharmatimes.com/Article/12-10-12/Goldacre_takes_ABPI_to_task_over_book_snub.aspx )*
Those who want to see Ben Goldacre's approach to systematic reviews should have a look at P35-36 of Bad Pharma for his summing up of the results from randomised studies of editorial bias.
'Leading academics' is a phrase that leaves me cold. I thought that the whole point of being a journalist was that rank, whether academic or otherwise, meant nothing. I am a leading academic** in my very narrow field but in that field, if I used it as an argument that I was right, everybody would say 'Stephen has now gone gaga'.
Here's a paper by Rawal and Dean that I rather like http://informahealthcare.com/doi/abs/10.1185/03007995.2013.860371
Stephen Senn
*I must say it all gets very confusing what with Ben Adams, Adam Jacobs and Ben Goldacre but I have to give Goldacre a prize for having a surname that is not the plural of a first name.
** Of course, some who know me would say that I am a misleading academic.