Dodgy statistics on airport security

I wonder if I should start a "dodgy statistic of the month" competition? I've just seen a good candidate for the prize this month, namely a story that "Nine out of 10 British people are happy to use full-body scanners being rolled out at UK airports".

Oh really?

Surveys are, of course, crucially dependent on how you ask the question and the context you put it in (as well as how you choose your sample, of course). I suspect you'd get very different results depending on whether the actual question used was:

(a) Given the serious and increasing risks of catastrophic terrorist attacks on air travel from violent extremists, with the potential to kill and seriously maim millions of people, would you be in favour of increased security at airports as a valuable measure to help catch those evil terrorists?

or

(b) Do you want it to take even longer than it currently does to go through all those tedious security checks at airports and for airport security staff to see you naked?

I thought it might be instructive to see if the methods of the survey were publicly available. They are not. Given that, and that the survey was sponsored by a company called Unisys, who by a strange coincidence happen to be in the airport security business, I think it's probably fair to say that we shouldn't take this "9 out of 10" statistic at face value.

I have a good rule of thumb with statistics based on surveys. If the survey methods aren't published together with the results, then the results are unlikely to be worth the paper they're printed on.

Update 6:25 pm:

Since I wrote this blog earlier a helpful chap from Unisys has been in touch and provided me with the details of the survey. He also pointed out that the report can be downloaded from the Unisys website for anyone who wants to go through the registration process.

So, were my worst fears about the data justified?

Well, thankfully, the question certainly wasn't as bad as option (a) above. But it was probably a bit closer to that than to option (b). The question was "Which of the following statements describe your willingness to sacrifice some privacy for enhanced personal security and convenience when you travel by air?" One of the following statements was "Full electronic body scans at the airport". So respondents are subtly pointed in the direction of believing that body scans will provide "enhanced personal security and convenience" when they answer the question.

The context is also interesting. This question was asked after a series of other questions about security threats. So by the time respondents had got to this point, they were presumably already thinking about various threats, which may have had an effect on the outcome.

On the plus side, the selection of participants for the survey seems to have been done quite carefully to ensure that it was a random sample of the population.

This is more than could be said for a survey on the Guardian website, which found very different results, namely that only 27% of people were in favour of full body scanners. It's not clear how participants were selected for that, but I would guess by self-selection among those who happen to be reading the Guardian website, which is not even remotely a random sample of the population. The questions on the Guardian website were also rather loaded, with the against-full-body-scanning option worded "No. It's a pointless invasion of privacy."

Both surveys seem to be biased, one in favour of body scanners, and one against it. No doubt the truth is somewhere between 27% and 90% of people being in favour of full body scanners.

The moral of this story is that if you read a statistic that has come from some kind of survey, you need to know exactly how that statistic came to be.