Why it was infeasible for the phase 3 vaccine trials to use "all cause deaths" as the endpoint

I have seen numerous people claim that the placebo-controlled, phase 3 vaccine trials that used symptomatic infections as the primary endpoint should have used "all cause deaths" to demonstrate that the vaccines reduced all-cause deaths relative to placebo. Indeed, preventing deaths is a primary goal of any medical intervention, including vaccines, and in some medical areas like phase 3 randomized cancer trials, all cause death is the primary end point. It would certainly be useful for us to have information on relative rates of all-cause deaths in vaccination and placebo arms from a large, adequately powered randomized trial.

So why were the phase 3 vaccine trials not designed with "all cause deaths" as primary endpoint? Amongst other reasons, it was practically infeasible, requiring a study lasting many years while withholding vaccination from hundreds of thousands of people for all of those years. Here in this post, I'll briefly illustrate this point.

Let's think about designing a randomized trial in the USA to test whether vaccines reduce all-cause deaths relative to placebo.

To be conservative, let's assume we can accrue a study population representative of the entire USA population, with the proportional number of very vulnerable at highest risk of death (This is very conservative because it is likely that any trial population would have had a healthier demographic, since it is not typical to allow the most sick to participate in a clinical trial for a new intervention).

Assuming a background non-COVID death rate of 2.8m per year, the approximate number of annual deaths immediately pre-pandemic, in both the placebo and vaccine arms, that would correspond to 233,333 deaths/month, which with a population of ~330m would result in a death hazard of 0.000707 per month.

Assume that we would like to power the study to detect a significant difference assuming that the vaccines prevent 90% of all COVID-19 deaths, and does not cause any extra toxicity deaths. From June 2020 to June 2021, there were an average of 2.625m confirmed COVID-19 cases per month, which corresponds to a 0.8% case rate. Let's assume this is the rate of COVID-19 cases in the placebo group. If we assume a 0.5% case fatality rate for placebo, this would mean that the hazard of COVID-19 deaths is 0.008 x 0.005 = 0.00004 per month (1/50k) in the placebo arm. With a 90% reduction in COVID-19 deaths, we'd have a hazard of COVID-19 deaths of 0.00005 (1/500k) per month. Thus, the total hazard for all cause deaths in the placebo in this scenario, obtained by summing background deaths and COVID-19 deaths, would be 0.000707+0.00004 = 0.000747 per month

For vaccine, the total hazard for all cause deaths would be 0.000707+0.000004 = 0.000711 per month. This corresponds to a proposed hazard ratio of 0.000711/0.000747=0.952, showing a ~5% reduction in total all cause mortality per month.

Now, the Pfizer and Moderna phase 3 trials accrued about 11k subjects per month. Assume that is the amount we can feasibly recruit for this trial. To have 80% power to detect this level of difference, using a log rank test with 0.05-level two-sided significance level, the study would require 12,978 all cause death events, which would require 48 months to accrue 501,937 subjects (accrued at 10,457/month), and allowing the study to go 60 months before completion.

Thus, this study would take 4 years to accrue and 5 years to complete, and that is only if the pandemic continued with the same community infection rate and case fatality rate throughout the 5 year period. If the pandemic slowed or less lethal variants emerged, this would extend the study even longer, much longer. A study this long during the pandemic would not have been feasible. And this assumes that we are fine with having >250k people remain unvaccinated during the pandemic for 5 years even if the vaccine was showing promise, another infeasibility. If an EUA were given for any vaccine, it would not be medically ethical in the USA to withhold a publicly available vaccine from the trial participants on placebo. It is very difficult to use all cause death as an endpoint in a phase 3 study for any preventative intervention, since only a small and highly uncertain proportion of the population will ever get the disease, and the others do not count towards the efficacy outcome. This is one reason why this endpoint is infeasible for studies of chemoprevention and other preventative interventions in cancer and not used.

It would certainly be a useful study if it could be done, but there is a reason why reduction of infection risk is used as the primary endpoint in vaccine trials, and why we have to rely on post-approval population level studies to assess factors like all-cause deaths.

Fortunately, countries like the UK have publicly shared data on all-cause deaths split out by vaccination status that can be used for such assessments, although as with all observational data, the lack of randomization means we must account for any potential confounding factors to get an accurate estimate of the causal effect of vaccines.