The day began with a talk that was supposed to be an update about what NRES was doing. In fact it was mainly about NRES’s mission statement, the drafting of which seems to have occupied an outrageously disporportionate amount of NRES’s intellectual efforts. As I wrote on my feedback form for the event, this focus on writing a mission statment shows “an alarming lack of perspective”. Maybe I’m just old fashioned, but I’ve never really seen the point of any mission statement that doesn’t begin with the words “Your mission, should you decide to accept it…“. Did the Pharaohs have a mission statement when they built the pyramids? Did Nelson have a mission statement at Trafalgar? You wonder how much could have been achieved if all the effort that had clearly gone into the (as yet unfinished) mission statement had been spent on doing something useful instead.

That was depressing, if not altogether surprising. What made me really dispair for the future of research ethics, however, was that the talk was followed with some extremely lively questions from the floor about the wording of the mission statement. The number of ordinary research ethics committee members who seemed to think that the fine points of the wording of the mission statement was a sensible subject for discussion filled me with a terrible feeling that the world has gone into an inexorable decline from which it will never recover.

We then descended from this particular circle of hell to the next. This took the form of a lecture about the legitimacy of ethics review. I thought this sounded like it would be an interesting topic, but I couldn’t have been more wrong. The lecture was given by a philosopher, and  I do not have the faintest clue what he was actually talking about. Now, I don’t have any training in philosophy, so I can’t be sure whether my complete inability to discern a single coherent point from the talk was due to my lack of familiarity with philosophical concepts or was because he was talking complete bollocks. However, on talking to the delegate sitting next to me, who it turned out had an MA in philosophy and who also failed to perceive anything remotely comprehensible in the lecture, I felt I was probably starting to narrow down the options.

After an extremely welcome coffee break, we split into small workshops on various ethical topics. I went to one about a new system for “proportionate ethical review”. This is a new fast-track system for research applications which present “no material ethical issues” (at least in theory). In principle, it’s a good idea. Ethical review can be a slow and resource-intensive process, and for some simple, non-invasive studies, it may well be over the top. A pilot project is currently running within NRES to see how these low-risk studies can be reviewed more quickly. The only trouble is that the pilot doesn’t seem to have been designed to assess what seems to me to be the most important aspect of the scheme. Some ethical issues are rather subtle, but nonetheless important. It’s not uncommon that I’ll be in an ethics meeting and we will almost be ready to approve a project which seems completely reasonable to me and most other committee members, when one particularly alert member might point out some subtle ethical issue that none of the rest of us had thought about, and which we are then readily agreed needs attention before the project can be approved. The risk of the fast-track system is that such subtle issues may be missed. Nothing in the pilot scheme is designed to look at whether they are, which strikes me as a pretty major omission.

The day wasn’t all doom and gloom, however. I also attended an interesting workshop on the Human Tissue Act, which is an important piece of legislation in the context of clinical research. It’s complicated and confusing, but at least I’m not quite as confused now as I was before I went to the workshop. We also had some interesting group discussion exercises in which we looked at some specific ethical issues, which led to some lively and constructive discussions. It reminded me that ethics committees are fortunate to have a great many members who are prepared to give careful and intelligent thought to ethical issues.

Apparently, a group of prominent stem cell researchers have written an “open letter” to journal editors complaining that their excellent research is being blocked from appearing in prestigious journals by some kind of  peer review mafia. The “open letter” doesn’t seem to be very open, as I couldn’t find it anywhere on the web. Anyway, according to the BBC, the letter alleges that top journals such as Nature and Cell are using a clique of peer-reviewers in stem cell research who are deliberately blocking research by rival scientists. This is, of course, denied hotly by the journals who claim that their peer review process is entirely fair.

I find it impossible to judge the rights and wrongs of this case given how little information is available (in fact the only information seems to be from the BBC, who don’t link to any verifiable sources). Maybe the scientists who are complaining that their work hasn’t been published have simply not submitted research of any quality and are just whingeing, or maybe they have an entirely valid point and their excellent research has been unfairly blocked. I just don’t know.

There is, however, an entirely valid point behind all this, which is that the allegations about unfair peer review certainly could be true. Peer review is a deeply flawed process, which relies on human beings applying considerable skill in an perfectly fair and impartial manner. How likely is that to work all the time? Some peer reviewers (probably the majority in fact) certainly do an excellent job and give the benefit of their expertise to pass reasonable judgements on the papers they review and offer sensible and constructive criticism that helps papers be improved. However, some don’t.

One bizarre feature of peer-review, given that it is fundamental to the scientific literature and that science is based on empirical evidence, is that there is practically no evidence that it is effective. A Cochrane review published in 2007 concluded “At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research”, although they also noted that that should not be construed as evidence that it is ineffective.

What we do know, however, is that it is based on human beings, and that human beings cannot always be relied upon to act fairly and without bias. In fact one of the most fascinating pieces of research I have ever seen about peer-reviewed research, which is now over 30 years old, looked specifically at the effect of cognitive biases in peer reviewers. Mahoney reported a fascinating experiment in 1977, in which he gave 75 peer reviewers a fictitious paper to review. In all cases, the methods were identical, but different reviewers got different results. Sometimes the results agreed with the reviewer’s preconceived ideas about what the results should be, and sometimes they disagreed. Where the results agreed with the reviewer’s preconceived ideas, they rated the quality of the methods higher than when they didn’t. This is despite the methods being identical in all cases, and is a classic example of what psychologists call confirmation bias, namely the tendency to be more inclined to believe something that fits in with what you think you know already.

We write a lot of papers for publication at Dianthus Medical, so we get to see a lot of peer-reviewers’ comments. Most are fair and reasonable, but it’s not uncommon to find some that are really not. One particularly common bugbear is when a clinical reviewer starts criticising statistical methods, even though that reviewer is not qualified to do so and is usually wrong in their criticism. Although that’s annoying, it’s not usually a big problem for us as we simply explain why their criticisms are wrong and a reasonable journal editor will listen. However, I do worry about what a relatively inexperienced researcher without access to statistical support might do in such circumstances.

Peer review is held out to be the ultimate guarantee of the quality of scientific publications, but a closer look reveals that it really can’t be relied upon to be so consistently. In a way, peer review is a bit like democracy. Most people know that democracy is a lousy way to run a country (just look at how this country is being run at the moment!), but no-one has yet come up with an alternative that isn’t even worse.

Sounds like a worthy project.

However, I have some serious concerns, and I won’t be participating. My first thought when I received the letter was “how the hell did they get my name and address?” Apparently they got it from the NHS, who obligingly sent them my date of birth as well. This is, so they claim, all done in accordance with the Data Protection Act.

I have my doubts about that, but if there is really a loophole in the Data Protection Act that somehow allows the NHS to pass on my personal details to third parties, then shame on the UK Biobank team for exploiting it. I allow the NHS access to some of my personal data in the expectation that they will treat it in the strictest confidence, and I am deeply shocked that they have not done so. I don’t care how worthy a research project this is, I am now going to find it hard to trust them.

And if I were going to participate in the research, I would certainly need to trust them. They would hold a great deal of extremely sensitive data about me. As others have pointed out, there are some serious risks involved in that, both arising from the risk of a breach of internal data security in what is a very large project (ie with more opportunities for leaky data) and from the possibility that the government (which, as we all know, has a rather unhealthy desire to know about its citizens’ DNA) will be able to access the data if it decides that it wants to.

The tone of the invitation letter was, IMHO, also very inappropriate. They have already made an appointment for me. Although they make it clear that I don’t have to go to the appointment, the fact they made it in the first place is extremely presumptuous. If this had come before the research ethics committee of which I am a member, I expect we would have deemed that to be inappropriately coercive and would not have allowed it.

I also find it rather shocking that the ethics committee that approved the project thought it OK for the NHS to release personal data to a third party.

What was the ethics committee thinking?

Update 11:19 am:

And as if all that weren’t bad enough, I’ve noticed something else deeply worrying on the UK Biobank website. They say on their “Confidentiality” page that data will be stored anonymously.

That is not true.

In my ethics committee, we are always extremely strict about ensuring that researchers distinguish between anonymised data (in which it is impossible to link data to an identifiable individual) and pseudonymised data (in which the data are stored separately from identifiable information, but the link is still maintained and so the data can be attributed to an identifiable invididual by anyone who has access to the linking codes). The UK Biobank data will be pseudonymised, not anonymised, and for such as well-resourced study to misuse such ethically important terms, and be allowed to get away with it by their ethics committee, is staggering.

A hearty welcome to the team to Ruth from all of us at Dianthus Medical.

A previous Cochrane review had concluded the Tamiflu was effective in preventing the complications of flu, based on a meta-analysis of 10 studies that found a 59% reduction in hospitalisations. However, that meta-analysis, despite having been published in the prestigious Archives of Internal Medicine, contained a schoolboy error in its statistical analysis. This is surprising, given that Roche has some very smart statisticians and that one would hope that a journal such as the Archives would have some good peer-reviewers. However, it’s not that surprising. Mistakes like that get through more often than we’d like to think.

So what was the error? This is a bit technical, so if you don’t like stats, feel free to skip this paragraph. What they had done was add up all the hospitalisations in the Tamiflu and placebo groups in all 10 trials, and treat the totals as if they had come from a single trial. That is not a good thing to do, because the trials had different inclusion criteria and therefore different risks of hospitalisation, and not all trials had equal numbers of Tamiflu and placebo patients. The effect of the drug is therefore confounded by the type of trial. A correct way to do the analysis would either be by logistic regression, controlling for the trial, or by a meta-analysis of the results of all trials. But not by pooling the data and ignoring which trial they came from, which is what they did.

So, when the Cochrane reviewers came to update their meta-analysis, they realised that they needed to get more details on the 10 trials in the meta-analysis. Sadly, only 2 of them had been published. Now I don’t want to be too hard on Roche for that: the trials completed about 10 years ago, and at the time it was quite common for many trials to remain unpublished. So the sensible thing for the Cochrane reviewers to do would be to ask Roche to supply the data.

However, rather than asking Roche directly for the data, they went to Channel 4 News, who then approached Roche to ask for the data. That seems pretty extraordinary to me. There has been no explanation of why they went down that route. As Roche said in their response on the BMJ website, this was “a move that questioned whether the motives for inquiries were truly for clarity and scientific validation”. Hard to argue with that. I’d love to know why the Cochrane reviewers and the BMJ thought it appropriate to involve a TV company in a systematic review.

However, although Roche had a great opportunity at this point to occupy the moral high ground, they have, IMHO, squandered that opportunity by not making the data available in full. They were prepared to supply the data to the Cochrane reviewers if they signed a confidentiality agreement, but the reviewers were not prepared to sign such an agreement. This makes both sides look pretty bad to me. I don’t see why Roche can’t make the data available in full, and I don’t see why the Cochrane reviewers should refuse to sign a confidentiality agreement. Roche obviously believed that there were some valid reasons to keep the data confidential. It’s really hard to imagine what those reasons could be, let alone how they could trump the absolutely pressing PR reasons for making all the data available. It’s also hard to imagine why the Cochrane reviewers felt unable to sign the agreement, even if they had every right to feel a bit miffed at being asked to do so.

Nonetheless, some data were supplied, and although it wasn’t sufficiently detailed to answer all the Cochrane reviewers’ questions, progress was being made, dialogue had been established, and it might be reasonable to think that ongoing dialogue would result in the necessary data being supplied before too long. However, the Cochrane reviewers were too impatient for this. They decided to go ahead and publish their review anyway.

This seems pretty extraordinary to me. The whole point of Cochrane reviews is that they are supposed to be systematic, in other words to include all the available data. To knowingly publish a review that excludes 8 relevant studies because they weren’t willing to wait until they had got hold of the data seems extremely irresponsible.

They could have waited until they got the data.

Roche could have published the study reports, in full, on their website.

However, as it is, neither side did the things that they could easily have done to give us a reliable answer to the question of whether Tamiflu prevents complications of flu. So the rest of us still don’t really know whether or not it does.

All in all, a bad day for science.

I think this is probably because there is huge demand among medical writers to improve their understanding of statistical techniques. Writing about statistics is probably one of the hardest bits of medical writing, and I suspect that the popularity of the course had something to do with that demand for statistical training. I had discussions with quite a few people about other statistical workshops that EMWA should be providing, and I suspect that it won’t be too long before I have been talked into developing more statistical workshops for EMWA. That’s no bad thing. If I can do my little bit to improve the average quality of medical writing in Europe, then that’s something I shall feel good about.

Adam at the end of the Bexhill half marathon

So I was delighted when I finished in 2 h 3 min 26 sec.

The word “Poppy” in the title, is of course related to the Poppy Appeal, and I was therefore raising money for that charity. If you haven’t yet sponsored me for the race, there is still time to do so by visiting my sponsorship page. When so many of our troops are still dying on active service, the Poppy Appeal is as relevant now as it’s ever been, so please give generously!