Ben Goldacre's Bad Pharma

Anyone who reads this blog is almost certainly familiar with Bad Pharma, the latest book by Ben Goldacre. I’d like to share some of my thoughts about this book.

For those who aren’t familiar, despite the title “Bad Pharma”, the book presents Goldacre’s views on how the entire system of medicine is dysfunctional. He has harsh words not only for the pharmaceutical industry, but also for drug regulators, doctors, medical journals, and even medical writers. The first sentence of the book is “Medicine is broken”, and the rest of the book is devoted to explaining why Goldacre believes that to be true. We are told that drugs are tested in badly designed trials, that the results of those trials are misrepresented, and that doctors make their prescribing decisions based on flawed information.

So is Goldacre right to claim this?

Well, partly. He certainly makes some good and important points. He identifies some genuine problems with the way medicine is practised. For example, he describes at great length the problem of publication bias. Publication bias is what happens when not all the studies that are done on an intervention get published, and the ones that get published have more favourable results than the ones that don’t. It is indeed an important problem. He also is quite correct to criticise the secrecy surrounding the drug regulatory process. Regulators make decisions about drugs behind closed doors. Goldacre would like to see all the information on which those decisions are based, and the reasoning for the decisions, made public. It is hard to see how any reasonable person could disagree with that, and I absolutely agree with Goldacre that the secrecy surrounding drug regulation is scandalous.

However, Bad Pharma is not without its flaws. There are many places where it veers away from a scholarly investigation and comes across more as trying to sell a story. Goldacre frequently exaggerates the extent of the problems he describes, and ignores evidence that contradicts the story he is trying to tell. Goldacre has two jobs: he is both a scientist and a journalist. The job of a scientist is to search for the truth in a dispassionate manner, taking account of all relevant evidence, whether or not it is consistent with pre-existing beliefs. The job of a journalist, on the other hand, is to tell a compelling story, and not to worry too much about inconvenient facts that may spoil the story. I am left with the distinct impression that Goldacre wrote this book wearing his journalist hat.

This is disappointing. Goldacre tells us near the beginning of Bad Pharma about the importance of considering all the data on a particular subject, and not simply cherry-picking pieces of data that happen to support a particular argument while ignoring disconfirming ones. Goldacre should indeed be very familiar with this concept: he has thoroughly earned a reputation as a razor-sharp debunker of pseudoscientific nonsense. And yet sadly, cherry-picked statistics are not hard to find in this book.

We don’t have to look very far: there is one on page 2. Goldacre cites a 2007 study by Bero et al which investigated whether papers sponsored by the pharmaceutical industry were more likely to report results favourable to the sponsor than independent papers. In their primary analysis, they found no statistically significant difference between the two. However, in a secondary analysis, they did find a difference, with an odds ratio of 20. It’s that secondary analysis that Goldacre presents, and he also presents that statistic by saying that “industry-funded trials were twenty times more likely to give results favouring the test drug”, which is to misunderstand what an odds ratio is. An odds ratio of 20 does not mean that something is 20 times more likely.

Now, to be fair, Goldacre does describe a systematic review answering the same question on the next page. But would a more honest approach not have been to go straight into the review without reporting some scary but almost certainly misleading statistics first? As an aside, a new systematic review looking at the question of whether industry sponsored studies are more likely to be favourable to the sponsor’s product has been published since Goldacre wrote his book, and it seems that the answer to this question may have been affected by publication bias, as I blogged about recently. So while it may be true that industry sponsored studies are more likely to find favourable results, the evidence to support that claim is nowhere near as robust as Goldacre says it is.

There are places where Goldacre relies on papers that present very shaky evidence. If they were not confirming his pre-existing beliefs, I’m sure he would be among the first to debunk them, but this doesn’t seem to happen in Bad Pharma. I’ve blogged before about one particular example that Goldacre cited as fact, but which really doesn’t stand up to critical analysis.

And then there are places where the book is just factually wrong. For example, we are told on page 59 that in the context of reporting adverse events of drugs, pharmaceutical companies “only have to tell the regulator about side effects reported in studies looking at the specific uses for which the drug has a marketing authorisation”. And on page 61, we are told that there is no obligation to tell European regulators about adverse events observed in trials conducted outside the EU. These assertions are just plain wrong. Pharmaceutical companies are required to report adverse events from any use and from any country. To quote from page 7 of the European Medicines Agency’s guideline on Good Pharmacovigilance Practice, “The integrated benefit-risk evaluation should be based on all authorised indications but should incorporate the evaluation of risks in all use of the medicinal product (including use in unauthorised indications).” The EMA also make it clear (in their Q&A document, in answer to question 1) that the periodic safety update reports that pharmaceutical companies are obliged to submit must include data on the “worldwide safety experience”.

One thing I have found disappointing in reading responses to Bad Pharma is that some people who should know better have taken everything written in the book as established fact. The book certainly creates the impression of being well evidenced, as it has an impressively long list of references which supposedly support the points being made. However, merely citing a long list of references isn’t enough: closer inspection of those references often shows that they don’t provide the support for Goldacre’s points that he would like us to believe they do. Although one or two blogposts (that last one in Dutch, but Google Translate does a reasonable job on it for those of you whose Dutch is a bit rusty) have been written about Bad Pharma with a suitably skeptical eye, the shaky evidence base for the claims made in the book seems to have escaped the notice of many, including two of my personal heroes when it comes to debunking bullshit claims in medicine, David Colquhoun and Richard Smith. I think that’s a shame: a good scientist should treat all claims with skepticism and require that they be proven before they are believed.

The response from the pharmaceutical industry has also been disappointing. The ABPI’s response has been widely criticised as being inadequate. They appear to be denying that problems exist, rather than engaging constructively to try to fix the problems. It’s true that some of the problems Goldacre claims exist actually don’t exist, but this does not mean that no problems exist. I wonder if this polarisation in responses to the book is a consequence of the exaggerated claims made in the book, and whether a more balanced and evidence-based approach by Goldacre would have led to some more constructive responses from the industry.

But perhaps the biggest problem with this book is that it gives ammunition to pedlars of quack alternative treatments. Claiming that the entire system of conventional pharmaceuticals is broken is exactly what those quacks want to hear. Real harms can result from giving them an excuse to claim that their systems of alternative medicine are better than pharmaceuticals, as Professor David Smith has explained in this thought-provoking video response to Bad Pharma. It’s also not hard to find reviews of the book by quacks who seem to be using it for exactly that purpose. I’m sure that Goldacre does not approve of Bad Pharma being used to promote quack remedies, but surely that was a predictable consequence of writing such a one-sided account of the pharmaceutical industry?

That’s all I want so say by way of an introduction. In the coming days I shall be starting on a series of blogposts in which I shall look at some of the claims Goldacre makes in Bad Pharma in more detail, chapter by chapter, and giving my take on which of them stand up to scrutiny and deserve our attention, and which ones don’t. In the meantime, your thoughts, dear reader, would be most welcome via the comments form below.