Clinical trials and supermarket beef lasagne

I've been thinking about how clinical trials have certain things in common with supermarket beef lasagne.

No, don't worry, we don't usually find horses hiding in inappropriate places in clinical trials. But in both cases, the key to having confidence in the product is traceability. There is a long chain of things that happen between where you start and where you end up, and knowing what happens at each stage of the process is key to verifying the authenticity of the final product.

In a clinical trial, the raw material is various observations made on the patient who takes part in the trial, and the final product is a published paper describing the results of the trial. There are a great many stages between the two.

During a clinical trial, an investigator will see the patient, make various observations (for example measuring the patient's blood pressure), and record those observations in the patient's medical notes. Those notes will then be transcribed into the data collection instrument for the study, either a paper case report form or an electronic data capture system. If they are written on a paper form, then someone else in a data management department will enter them into a database.

So now we have data in a database. It's possible that errors may be discovered in the data once the data managers run various consistency checks on it, and those errors need to be corrected. At the end of that process, we have what we hope is a reasonably reliable database.

That database is then exported into datasets that can be used for statistical analysis. These will first be in the form of SDTM datasets: useful for storing data, but not necessarily all that useful for analysing it. Before we can analyse the data, we need to convert the SDTM datasets to analysis datasets.

We can then run our analyses. Those analyses will result in some statistical output. The medical writer will take the statistical output and use it to write a clinical study report in which the results are described in great detail. Finally, either another medical writer or the same one who wrote the report will take the clinical study report and use it as a basis for writing a paper for publication. That paper will then enter the public domain.

As you can see, there are a lot of steps in that process. It's really important that we can be completely confident of each step. To achieve that, we have generally accepted standards for traceability that are really very rigorous.

Independent monitors will routinely visit investigators and check that the data they have entered on case report forms match the patient's notes. If we are entering case report forms into a database, we do it in duplicate, so that data entry errors are minimised. We also do further quality checks to make sure that case report forms match the contents of the clinical database.

And if changes are made to the clinical database when errors are found? Those changes are also carefully recorded: databases conforming to industry standard practice for clinical data management keep detailed audit trails, so if data are changed then we know exactly what was changed, by whom, when, and why.

When we have got as far as our analysis datasets, we have detailed standards for traceability of the data, so each data point in our analysis datasets can be traced back to an individual data point in the clinical database. In fact, that's an exercise we frequently undertake as part of our routine quality control procedures when analysing clinical trial data. If we need to verify that the analysis correctly reflects the data, then our analysis datasets are what are used in the analysis, so it's a simple matter for another statistician to replicate the results (again, something we frequently do in practice as part of our quality control).

The clinical study report will contain the statistical output as appendices, so it's easy enough to verify that the text of the report is an accurate reflection of the analysis results. And when we write manuscripts for publication, we routinely QC them against the study reports, so we know that the manuscript is accurate.

As you can see, this is a long chain of events, but rigorous traceability standards are built into every step, so we can have confidence in the final results. These procedures are all thoroughly documented, so if one of our clients or a regulatory authority want to check what we've done just to make sure that it's really as traceable as we say it is, then they can.

It seems to me that this is how things ought to work when you go to the supermarket to buy a beef lasagne, but clearly it doesn't. What we've learned recently is that supermarkets actually have no idea what's in the products they sell. Any mechanisms for traceability from the animals that we think we're eating to the product on the supermarket shelves have failed. If supermarkets want to be confident that they're selling what they say they're selling, then they need to have a good look at how their traceability mechanisms work and come up with something a bit more robust. I think perhaps they could learn a thing or two from clinical research.

Here's a final, somewhat unsettling, thought about supermarket beef products. We know that a great deal of testing is currently going on looking for horse DNA (and some testing for pig DNA in products not labelled as containing pork). However, as far as I've been able to determine from both the Food Standards Agency and Tesco, no-one is testing for other species that shouldn't be there.

We know that attempts to know what is in processed food via traceability of ingredients has failed on a large scale. We are not looking at an isolated incident of one particular product containing horse meat: many different products from many suppliers are affected. It seems that this has only come to light after specific tests for horse DNA were done. Given that there is clearly a widescale problem with supermarkets not knowing what's in the products they sell, is it plausible that horse meat is the only unauthorised ingredient?

I suspect that this food labelling scandal could get worse, but we're not going to know just how much worse until people start testing for the presence of other types of DNA. To give one example, until someone has tested a beef lasagne for rat DNA, how can we know there is no rat in it?