New Lancet policy on systematic reviews

The Lancet have recently introduced an interesting new policy. They now require anyone submitting the results of research (not just randomised trials, apparently, but all research) to The Lancet to include a systematic review with their research. This can be a reference to a recently published systematic review, but if no such review exists, then the authors are required to do their own systematic review and report it within the paper.

Now, the intention behind this idea is entirely laudable. We all know (we do know this, right?) that the results of a single clinical trial are usually meaningless without being put into context. If I report a trial of an intervention that finds it is effective in a certain condition, then you might be inclined to believe the intervention is effective. However, if 25 previous high quality trials of the same intervention in the same condition had found it to be useless, I hope you wouldn't take too much notice of my one positive trial, which in those circumstances would look like a bit of a fluke.

So, by including the results of a systematic review in a research paper, the journal will make it much easier for readers to judge the reliability of any conclusions drawn in those papers, as a result of being able to see those results in their proper context.

There is, however, a catch.

A systematic review, if it's going to be done properly, is not a trivial exercise. Obviously it depends on the intervention being studied and how much is already known about it, but it can sometimes take several months to complete a systematic review on a subject for which there is already copious literature. This will dramatically increase the costs of preparing the paper.

Now, put in the grand scheme of things, that shouldn't matter too much. The costs of writing the paper are normally utterly trivial compared with the overall costs of doing the research that the paper describes, and even if the costs of the paper increase tenfold (which they easily could), it probably doesn't have too great an impact on the overall costs of the research project.

Of course, it's also good practice to do a systematic review of a subject before starting a new trial, to be sure that the trial is really needed, but in practice that isn't always done, and even if it is, then it might well be out of date by the time the research has finished and the paper is being written.

On balance, I approve of what The Lancet is proposing (declaration of competing interests: sometimes people pay us to write systematic reviews, so this could be an extra source of business for us), but if it is to be successful, then research funders will need to change the way they think about writing up their results. Although the costs of writing up research are tiny compared with the costs of doing research, the fact is that budgets for writing the research are usually fixed and small (or even non-existent), and if that remains the same, including the systematic reviews is going to be impossible. Research funders will need to understand that writing up the research for publication is important, that even with a systematic review it is a very small part of the overall budget, and that it is something that needs to be adequately funded if it is going to be done in a way that will advance medical science to the high standards that The Lancet are seeking to achieve.

One interesting little thing to note in closing: the Centre for Reviews and Dissemination are planning to start a prospective register of all systematic reviews. This is certainly an excellent idea, which should help to reduce bias in systematic reviews. However, I'm not sure whether they intend that all "built-in" systematic reviews in Lancet submissions should be registered.