When statistics lie, Statistics can unravel the lie

Most of you will know the saying “There are three kinds of lies: lies, damned lies and statistics”^*.  I regulary hear it when introduced as a speaker for a lecture to a non-statistical audience. Works well as a start for me: lesson one then is to explain the difference between statistics and Statistics (with a capital ‘S’). The first being the numbers that summarize data, the latter being my expertise. And yes, statistics may lie, but Statistics doesn’t. It may be abused to cover a lie, but in itself it is just a collection of mathematical techniques to analyse data. And math doesn’t lie. Lying statistics are everywhere these days, with the sad war in Ukraine as a prominent example: the number of reported casualties on either side depends strongly on who reports those. One golden propaganda rule is evident: always report an ‘exact number’ such as 3118, instead of 3000. You always want to give the impression that the number is an actual measurement, and not an estimate. I’d say Statistics can do little to counter the abuse of statistics as propaganda in a war. But in other situations, Statistics can certainly help unraveling lying statistics. I start with a historical example.

Mendel-Fisher controversy^$

Gregor Mendel is a founding father of the hereditary laws. In the 1860s, his large scale experiments with pea plants demonstrated the principle of dominant and recessive alleles. In short, his law states that the first generation offspring of parents that both have one dominant and one recessive allelle will possess the dominant characteristic (e.g. a long stem) in a 3:1 ratio w.r.t. the recessive one (e.g. a short stem). Why? Peas – and we as well – receive one allele from both parents. Suppose x is the dominant allele and o the recessive one. Now, if all combinations are equally likely then a child has either xx, xo, ox or oo. The first three lead to the dominant characteristic, a long stem, the latter to the recessive one, a short stem. Mendel was very efficient in breeding the pea plants, which is why he could grow large numbers. The statistics were very convincing, such as a 6022:2001 ratio for the dominant color of the peas. Almost too good to be true?

Years later (in the 1930s) another founding father, geneticist ánd statistician Ronald Fisher, demonstrated that this could well be the case. Fisher used a clever trick: he assumed that Mendel’s theorem is true (in fact, Fisher also believed that) and computed the probability that under random sampling one would observe a ratio at least as close to the truth (3:1) as the one reported by Mendel. Fisher’s initial analysis rendered a probability 1/16. Small, but probably not small enough for firm conclusions.  Later, however, Fisher presented an even stronger case. Of those pea plants with dominant characteristics 2/3 should be heterozygote: one x and one o, and 1/3 homozygote: both x. Hence, a 2:1 ratio of heterozygote vs homozygote. Mendel presented statistics extremely close to this ratio, namely 399:201. This is already very unlikely if 2:1 is the truth. Fisher, however, discussed that in fact Mendel’s tests for heterozygosity are imperfect and in fact around 5.6% of these are classified as homozygous. But that means that the ‘true’ ratio is in fact (2 – 0.056*2)/(1 + 0.056*2) = 1.88:1.11. And then, of course, it is even more unlikely to observe a ratio at least as close to 2:1 as 399:201 is. In fact, when aggregating results from several experiments, Fisher computed a probability as small as 0.00007. That is really small. Some, however, believe that this is either coincidental, or that there are botanical explanations. The controversy on whether Mendel’s results were forged or not went on for years. See here^$ for a nice, extensive exposition. Of note, Fisher never blamed Mendel for forging results, but went for the easy target: Mendel’s assistents.

A contempory example: the Sputnik vaccine^#

A colleague of mine used a similar Statistics technique to indicate suspicious statistics in the original Lancet publication on the efficacy of the Sputnik vaccine. The original publication reported very homogenous efficacies of the vaccine across five age groups, all close to 91%. These are important statistics, because other vaccines usually showed a decreasing efficacy over age. Now, my colleague applied a dedicated Statistical test to quantify whether the reported statistics are suspiciously similar, even if homogeneity would truly hold. So, basically the same trick as Fisher used: show the numbers are too good to be true, even under the most favorable assumption. Remember, suppose the true efficacies are all 91% (hence completely homogeneous), we still expect the measured ones to fluctuate and more so when the sample sizes are fairly small. My colleague computed that the chance to observe homogeneity at least as extreme as indicated by the reported statistics is 0.0037, so 0.37%. Hence, rather small. Not a formal proof of forgery, but along with the other reported criticisms^# on the vaccine study, at least suspicious.

A word of caution

Some statistics are really hard for Statistics.  If one investigates multiple statistics, one has to account for this multiplicity: the probability that at least one of those seems suspicous by accident increases with the number of tests.  Also, the statistics may not be independent. A strong dependency may render the observed statistics much less extreme as they would be under independence. Think of a random sequence of 0’s and 1’s, each occurring with probability 0.5. Then, observing 1111111111 is rather unlikely under independence, but much less so if the probability of a 1 is increased a lot when the previous number was a 1. Finally, the posthoc fallacy is on the lure: once one suspects a lie, one may not be open for other likely explanations of the suspicious statistics. So yes, Statistics can certainly help unraveling statistical lies, but should be complemented by other evidence to substantiate the proof.

Notes

*The saying is often attributed to Mark Twain, but the actual originator is not clear, see Wikipedia

^$Allan Franklin provides an extensive overview of the controversy

^#More details, including other criticisms on the Sputnik vaccine study, are published as a letter to the Lancet

Main image: Niebieskibanan on Pixabay