GREAT News about SARS-CoV-2 Vaccine from Pfizer/BioNtech! Some statistical perspective.

Updated: 5 days ago

Pfizer has a press release on an interim analysis from their SARS-CoV-2 vaccine trial. Here is a link to an NPR article on the release. Here is an article discussing this in Nature

This is exciting news since early results providing strong evidence of efficacy and safety could lead to an emergency use authorization (EUA) by the FDA, and potential distribution starting the wintertime. The possibility of this timeline has been discussed for many months, and if these results pan out, then indeed it could be realized.

The article mentions that the preliminary analysis found the vaccine to be "90%" effective, but it is not clear what this means. The trial had 43,538 participants randomized to placebo or vaccine, and at the time of the interim analysis 94 participants had tested positive for SARS-CoV-2, all symptomatic. The way the analysis works on this type of trial is to assess whether the active vaccine group had considerably fewer testing positive than the placebo.

UPDATE: From the protocol, page 30, here is the description of the primary endpoint:

In participants complying with the key protocol criteria (evaluable participants) at least 7 days after receipt of the last dose of study intervention: 100 × (1 – IRR) [ratio of active vaccine to placebo]

At a minimum it likely means a 90% reduction in expected cases based on the placebo group -- this would correspond to 86/94 from placebo group, and 8/94 from the vaccine group, suggesting a >90% reduction of the expected infections in the vaccine group from 86 down to 8. 1-8/86=0.907. This is indeed promising and would provide clear statistical evidence that the vaccine reduced infection rates relative to placebo. The p-value would be <0.00000000000000001, so would be incontrovertible evidence that the vaccine prevents infections.

It is also possible that it means that the lower 95% confidence limit is 90%, showing statistical evidence that it is at least 90%, which would require a ratio of cases from vaccine to placebo group of 3:91, which would be an observed efficacy of 96.7%, and would have a p-value of <0.000000000000000000000001, even smaller.

For perspective, the flu shot has about 50% effectiveness, meaning it prevents roughly half of the infections that would have occurred sans vaccine, and the measles shot is 93% effective based on the first shot and 97% effective based on the second. So if this holds up in the population, 90% effective would be PHENOMENAL and more than anyone dared to hope for. If it was 96.7% then its efficacy would be equivalent to measles. This is also historic because no mRNA-based vaccine has been shown to work like this before, which also gives hope that Moderna's mRNA based vaccine may work as well.

This is promising, but even then there would still be questions about how well it does in various subgroups, especially high risk groups, and also about how its performance might vary based on amount of inoculum, i.e. the volume of virus to which one is exposed, i.e. whether it activates the immune system quickly enough to prevent significant infection if exposed to a high volume of virus. It is also possible that the vaccine is not preventing infection, just suppressing symptoms, and that some of those whose infections were "prevented" by the vaccine are still asymptomatic carriers. All of these subgroup analyses will surely be done with these data even if not in the press release, and I hear that they will do a safety comparison of the two arms in the coming weeks, but the trial would have stopped early to check had there been any safety issues among the >42k participants in the trial so it is unlikely there is anything significant. Analyses will also be done to assess the efficacy after just one dose to see how much protection is conferred while waiting to do the second dose.

Also, a safety analysis comparing placebo and vaccine arms will be done when all subjects at this interim analysis are at least 2 months past second dose, which should occur in the next couple of weeks.

Another caveat is that this vaccine has to be given in 2 doses, and requires storage at -94 degrees Fahrenheit, so distribution would pose significant logistical challenges. Note that Moderna's mRNA based vaccine only requires storage at -4 degrees Fahrenheit, so if that vaccine also seems to work, the logistical problems would not be nearly so great.

This is good news, but I will try to hold back getting too excited until I have a chance to see the details of the efficacy and safety data to be sure it is indeed as promising as portrayed. But I have to admit I am pretty stoked and can see a path for us to move past this pandemic in 2021.

This is probably the best news we have received from science during this entire pandemic.

By the way, just today (November 13th) my fellow biostatistician Natalie Dean and epidemiologist Mark Lipsitch, two of the top experts in vaccine trials, have published an outstanding article in Science describing how to understand COVID-19 vaccine trial efficacy. I highly recommend it.

Also, here is a follow up article from Wall Street Journal that discusses next steps and potential timelines:

UPDATE (8/18): The BioNTech/Pfizer vaccine study whose interim results were presented last week showing results after the first 94 events has now completed and reached 170 infection events. Their press release this morning reported 95% efficacy, which is the same as Moderna's. The corresponding 95% confidence interval on efficacy would be (0.895, 0.978).

These results are curious if the interim analysis truly had 90% estimated efficacy, since in that case all 76 of the infections since the interim results would have been in placebo, but maybe there is another explanation such as the interim analysis efficacy was >90% but the DSMB only reported that it was >90% without revealing the exact number, or that the lower confidence band of the efficacy result was >90%, but whatever the case the results are impressive and promising.

Additionally, they reported 9 severe cases in the placebo arm and only 1 in the vaccine arm, plus relatively mild adverse events with fatigue (3.8%) and headaches (2%) the most commonly reported symptoms.

Some additional calculations and projections: Here are some additional statistical calculations about what to expect in the final analysis that will be done after N=164 events:

We can use something called "Bayesian predictive probabilities" to compute the probability of various outcomes at the end of the study based on results from the interim analyses, assuming here to be 8 events for vaccine arm and 86 for placebo, which is likely the worst it could be based on the press release. These calculations are easily done using a "Beta Binomal Distribution".

Based on these calculations, the probability that, at the end of the study at 164 events, ...

  • Probability (estimated effectiveness is at least 50%) = 1.00

  • Probability (estimated effectiveness is at least 80%) = 0.9993

  • Probability (estimated effectiveness is at least 84%) = 0.99

  • Probability (estimated effectiveness is at least 86%) = 0.95

  • Probability (estimated effectiveness is at least 90%) = 0.65

That is, it is certain that the study will meet the FDA requirements, which means that the FDA will have enough evidence for the minimum efficacy endpoint. And it is all but certain that the ultimate estimate of effectiveness is at least 80%, and almost certain to be greater than 85%, and quite likely to remain above 90%.

For anyone interested in the technical details of these calculations, here are my assumptions:

  • The final analysis is done at 164 events (infections) as proposed

  • The interim analysis had 8/94 events in vaccine group and 86/94 in placebo group

  • Assuming a Uniform prior on the proportion of events in vaccine group at the beginning of the study, which means a Beta(9,87) posterior after the interim analysis.

  • With 70 events remaining to be observed to complete the study, one can can compute the Probability (X1<=x), where X1=number of the 70 events in the vaccine group, using a Beta-Binomial (N,a,b) distribution with N=70 and parameters a=9, b=87.

Following are the values of x that yield the effectiveness values mentioned above:

  • 50% effectiveness: 55:109 ratio of vaccine to placebo cases, which would require 47:23 ratio in second half of the study which corresponds to x=47

  • 80% effectiveness: 27:135 ratio of vaccine to placebo cases, which would require 19:49 ratio in second half of the study which corresponds to x=19

  • 84% effectiveness: 23:139 ratio of vaccine to placebo cases, which would require 15:55 ratio in second half of the study which corresponds to x=15

  • 86% effectiveness: 20:144 ratio of vaccine to placebo cases, which would require 12:58 ratio in second half of the study which corresponds to x=12

  • 90% effectiveness: 15:149 ratio of vaccine to placebo cases, which would require 7:63 ratio in second half of the study which corresponds to x=7

Note that these probabilities would be even greater if the interim result had ratio of cases in vaccine to placebo less than 8:86, which might be the case, so these results are conservative.


©2020 by Covid Data Science. Proudly created with