HIV vaccine results

Today's big health news story is a "breakthrough" in HIV vaccine research, as the results of a study done in Thailand are announced. At the end of a 3-year study, 74 of 8,198 subjects became infected with HIV in the placebo group compared with 51 of 8,197 in the vaccine group. That's a vaccine efficacy of about 31%, or if you prefer, a risk ratio of 0.69.

Well, on the face of it, that does sound like a breakthrough, doesn't it? But if I were an HIV researcher, I don't think I'd be opening the champagne just yet*. Most of the mainstream media is reporting that this result is statistically significant. Indeed it is. P = 0.048 by my calculations (Fisher's exact test). That is only just on the right side of the totally arbitrary value of P = 0.05, taken as statistical significance. In other words, there's about a 1 in 20 chance that these results could have been obtained by chance even if the vaccine were totally ineffective.

To put it another way, if just one more person in the active treatment group had become infected with HIV, the results would not have been statistically significant (P = 0.056, if you must know). There's always something slightly suspicious about P values like 0.048, as you wonder if some statistical fiddling has been going on to get the P value just on the right side of 0.05. It would be easy enough to exclude just one HIV-infected, vaccine-treated subject from the analysis on the grounds of some protocol violation or similar. Normally, that wouldn't make any difference, and probably doesn't make much difference here either, except that a P value of 0.048 will be widely reported in the popular media as statistically significant, whereas a P value of 0.056 won't. Of course, I have no reason whatever to think that the researchers here have manipulated their results in any way, but as the research doesn't appear to have yet been published in a peer reviewed journal, nor do I have any  way of satisfying myself that they didn't.

Let's hope that the vaccine was indeed effective (vaccine researchers will have to do considerably better than 31% efficacy if they are going to come up with anything clinically useful, of course, but the proof of concept would nonetheless be hugely important), but I'm not convinced just yet. These results desperately need replicating in further studies if they are to be believed.

As an aside, I'd also like to know more about the ethical aspects of this study. We are told that study subjects were at average risk of developing HIV and that all were given counselling on HIV protection and free condoms. But despite that, almost 1% of the placebo group became infected. Given that the prevalence of HIV infection among Thai adults is 1.4%, that 1% infection rate seems rather high to me. I do wonder how carefully the counselling on HIV protection was done.

* Actually, that's a lie. I like Champagne, and am quite happy to open a bottle at the flimsiest excuse. In fact, a piece of advice given to me by a seasoned researcher when I was a young PhD student back in the 1980s was that you should always open the Champagne at the merest hint that your research might have been successful, because if you wait until you have conclusive proof, you will never drink Champagne at all.