Always read the full paper

My attention has just been drawn to a paper which reviews basic research in homeopathy. It's bollocks, of course, as is most research in homeopathy, but it does provide a useful little lesson in the importance of reading the full paper and not relying on the abstract.

In the abstract, we are told that their literature search identified 830 basic science experiments (ie experiments in labs, rather than clinical trials) of high potency homeopathic dilutions. For those not familiar with the terminology, homeopaths use the term "high potency" to mean "highly diluted", to the point where it is extremely unlikely that a single molecule of the original substance remains.

Of those 830 studies, the abstract tells us that at least one positive result was reported in 90% of them.

Sounds impressive, doesn't it?

Well yes, I have to admit that it does sound impressive. But to see whether it actually is impressive, you have to read the full paper.

The first thing I'd want to know is what exactly do they mean by a "positive result". Do they mean a statistically significant result on the pre-specified primary outcome variable? Or perhaps they mean any statistically significant result on any of the (potentially many) variables investigated, which would be less impressive. Or maybe they even mean any result of any size in a hypothesised direction, whether or not it's statistically significant, which would actually be completely meaningless.

Which of those definitions did they use? Well, I don't know, because they don't say. However, I'm prepared to bet the price of a year's supply of 30C arnica that it wasn't the first of those definitions.

So, we know that 90% of the studies found at least 1 positive result, but now we know that "positive result" was not defined, which already makes it pretty much meaningless.

But it gets worse.

We are also told that they did not collect the information on whether the "positive results" were actually obtained from the "high potency" dilutions. Many of the experiments (we are not told how many) also used low potency dilutions, ie substances which have not been diluted so much, so that they may well have a pharmacologically active substance in them. So even if studies did have positive results, we do not know whether they came from the high potency or the low potency dilutions. So as a way of assessing the effects of high potency dilutions, this paper is about as useful as a bacon sandwich at a Bar Mitzvah.

What we are told, however, is that 4 studies found that high potencies caused significant effects in the opposite direction to low potencies, which is what we might expect if the "like cures like" principle of homeopathy were true.

That's 4 studies. Out of 830.

All of these details would be unavailable to someone who had just read the abstract. Anyone who read only the abstract would just read the headline conclusion that 90% of the experiments were positive, and not be aware of all the crushing limitations of that figure.

So the moral of the story is clear: never attempt to draw conclusions from a published paper based on the abstract alone.