Survivor bias cuts both ways and can be erroneously used to argue both for and against vaccine safety

Update: Here is a video with a graphical explanation:

Is attending the cinema deadly?

Or, to express the question statistically, does more frequent attendance increase or decrease mortality?

This sounds like a crazy question because intuitively we know that cinema visits have no impact on your health. But imagine that a study has been done that collected data on people who had died after visiting the cinema and concluded that attending the cinema contributed to an increase in mortality. Should you believe this conclusion?

We aim to show that there are two different types of survivor bias which, by suitable manipulation of the statistics, can be used to show both increases or decreases in mortality rates the more cinema visits that are made.

Using survivor bias to ‘prove’ decreased mortality with increased cinema visits

The first type of survivor bias is to simply count the number of deaths out of those who had 0 visits, 1 visit, 2 visits etc. This will inevitably result in those with the most visits having the lowest mortality rate because those who die during the period have less time to accumulate cinema visits than those who survive to the end.

To understand this, just imagine an experimental scenario in which we select 10,000 people who have never been to the cinema before. We monitor them over two equal duration time periods. After the first time period half the people attend the cinema once during the second time period and the other half do not. Let’s call these group A and group B, where group B are those who will attend the cinema in the second time period, assuming they survive to do so.

In this imaginary study the people are homogenous with similar demographic and health attributes.

Assuming that in any time period 2% of the 10,000 people die for any reason then, after the first time period, 200 will have died with 9,800 surviving. That means 4,900 of the survivors of group B get to attend the cinema during the second time period and 4,900 from group A do not (because they never could). In each of these two groups 98 die by the end of the second period.

The total number of people who attended the cinema in the second time period is 4,900 from group B. The total number of people who did not is 5,100 and is made up of 5,000 in the first period, from group A, and the 100 from group B who died before they could do so.

Thus, we observed 4,900 who attended cinema once, of whom 98 died (a mortality rate of 2%) and 5,100 who never visited the cinema at any time, of whom 298 died (a mortality rate of nearly 6%). This shows that the more visits the lower the mortality rate.

The error in this analysis is due to the fact that we are using the wrong measure for mortality rate. Instead of just dividing number of deaths by number of people in each category (no-visit or one-visit), we need to take account of the amount of time spent (number of person periods) in each category. The 4,900 people who had one visit each spent one period in the ‘no-visit’ category and one period in the ‘one-visit’ category. So, in the one-visit category there were 98 deaths in 4,900 person periods which is a mortality rate of 20 deaths per thousand person periods.

And in the first period all 10,000 people were in the no-visit category, while in the second period 4,900 were in the no-visit category. That means there was a total of 14,900 person periods for the no-visit category. And there were 298 deaths among these, giving us a mortality rate of 20 deaths per thousand person periods. Exactly the same as the rate for the one-visit category.

Using survivor bias to ‘prove’ increased mortality with increased cinema visits

However, survivor bias can be used to prove the opposite: that mortality increases with cinema visits.

In the previous imaginary study, we demonstrated that we need to consider mortality rate per unit time (person period) in order to avoid the survivor bias. But this still does not save us from another kind of survivor bias which inevitably occurs when the cohort under study does not remain homogenous over time, for example when it includes elderly people closer to death.

Irrespective of the number of cinema visits, during the period of observation more elderly people will have a significantly declining survival probability over the course of the study. To show how this works here we assume that during the first period the mortality rate is 2% but, for those surviving this first period, the mortality rate increases, to 3% in the second period.

Then, exactly as before we have all 10,000 people in the first period in the ‘no-visit’ category of whom 200 die. And in the second period we have 4,900 people in the one-visit’ category and 4,900 people in the ‘no-visit’ category. The difference now is that in the second period the mortality rate is 3%. This means that 147 people in each of the two categories die in the second time period.

So, in total there are 4,900 person periods in the ‘one-visit’ category with 147 deaths, giving a mortality rate of 30 per thousand person periods.

And there are a total of 14,900 person periods for the no-visit category with a total of 347 deaths, giving a mortality rate of 23 per thousand person periods.

Clearly the more people visited the cinema the higher the mortality rate.

However, clearly, this conclusion is simply an inevitable statistical artefact of a declining survival probability in an aging population.

What this means for studies into vaccine safety

While the above examples only compared the ‘no-visit’ mortality rates with the ‘one-visit’ mortality rates it is clear that the illusions extend to any number of visits.

Moreover, if we simply replace ‘cinema visits’ with ‘vaccine doses’ then it is clear that we can create - for a placebo vaccine - exactly the same statistical illusions of either increased safety, the more doses you take, or decreased safety, the more doses you take. For the first type of survivor bias we can also replace deaths with ‘cases’ to erroneously conclude increasing efficacy for the vaccine after each dose.

We don’t even need to have zero-doses as the control group to see these effects. If we have one-dose and two-dose this would show the same results as zero-visits and one-visit to the cinema.

We have previously reported how the first type of survivor bias has featured in studies claiming vaccine safety and efficacy (especially those relating to vaccines in pregnancy) despite the obvious way to avoid it.

However, it also possible that the second type of survival bias has featured in studies that have demonstrated increasing mortality with each additional dose of the vaccine. In such cases we need to be clear that the increasing mortality rate is not simply confounding the known effect of normal declining survival probability in an elderly population with a non-uniform mortality rate over the study period.

Where are the numbers? by Norman Fenton and Martin Neil is a reader-supported publication. To receive new posts and support our work, consider becoming a free or paid subscriber.

This article was originally published by Where are the numbers? and is republished here with permission