Last week, as I highlighted in __my blog post__, BioNTech/Pfizer released the interim analysis of their phase 3 study results for their SARS-Cov-2 vaccine, demonstrating 90% efficacy, meaning the vaccine prevented 90% of cases that would have occurred sans vaccine, using the most rigorous study design available: a double-blind randomized design in which no one knew whether each individual got vaccine or placebo until unblinded for the analysis. After enrolling >44,000, they found that out of 94 cases, a vast majority (~86) were in the placebo arm and much fewer (~8) in the vaccine arm, implying that ~78 of these cases were prevented by vaccine. I noted that that even those these sample sizes may seem small, they are not in this context, and the p-value for demonstrating efficacy is vanishingly small (p<0.00000000000000001), and the 95% confidence interval for efficacy was 80.8%-96.2%, both using basic binomial calculations.

**Limitations of BioNTech/Pfizer Results**

Two key limitations of these results and vaccine are (1) we didn't know whether it was effective in preventing severe disease, or its efficacy was limited to mild disease, or even worse exacerbated severe disease in the vaccine arm through something called "antibody mediated enhancement (AME)" which immunologists mentioned is a possibility, and (2) this vaccine requires two doses and storage at -94 degrees Fahrenheit, making storage and disbursement exceptionally challenging in a logistical sense.

**Incredible results from Moderna**

Well, the press release from Moderna this morning, as described in __this NY Times article__and this __NIH report__, found that the Moderna vaccine had even better results: 94.5% efficacy, with 95 total cases, 90 in placebo arm and only 5 in the vaccine arm. This efficacy rivals some of the most effective vaccines (e.g. measles 93% for single dose and 97% after both doses) and much better than influenza (40-60% efficacy), to which many people keep comparing SARS-CoV-2 in spite of the fact that there are known extensive differences between the two viruses.

Based on quick binomial calculations that assess the probability of seeing a 90/5 split if in fact vaccine and placebo subjects had the same probability of infection, the p-value for testing whether the virus has any efficacy is vanishingly small (p<0.000000000000000000002), the p-value for testing efficacy better than 50% or 75% is p<0.00000000004 and p<0.00005, respectively. The 95% confidence interval for efficacy based on these binomial calculations is 86.5%-98.2%.

This is pie-in-the-sky incredible results far better than anyone dared dream. As I mentioned, the FDA set the bar at 50% efficacy, both the Moderna and BioNTech/Pfizer studies had targeted 60% efficacy, and Fauci has said he was realistically hoping for 75% efficacy. Now, we have two separate vaccines, both using a never-before-used mRNA based technology that affords great flexibility in designing the vaccine to target the most robust and effective part of the virus, that have had >90% efficacy.

**Some key facts about the Moderna protocol**

Here is the __Moderna protocol__. From page 10 of the protocol, you can see that a case was defined as a positive PCR test after experiencing either two COVID-19 symptoms or one respiratory COVID-19 symptom. The study involves 30,000 subjects equally randomized to placebo and vaccine arms. The final analysis will be completed when 151 of the subjects have become confirmed cases infected with SARS-CoV-2, with the two interim analyses proposed after 53 cases and 106 cases. This analysis is based on 95 cases, so it is not clear whether this should be considered a late application of the first interim analysis or an early application of the second interim analysis.

Notably, 37% of participants were racial or ethnic minorities.

**Moderna vaccine prevents severe cases**

Unlike the BioNTech/Pfizer press release, Moderna's included some information in terms of prevention of severe cases. 11/95 of the cases were severe, and all were in the placebo arm. If you are interested in p-values here the p-value for assessing efficacy of the vaccine in preventing severe disease is p=0.0005 and for assessing whether its efficacy for preventing severe disease is at least 50% is p=0.01, which is notable given the small number of severe cases, 11. This provides considerable evidence that it is not just preventing mild disease, but severe disease, as well, and also with no severe cases in the vaccine arm, shows that at least in this cohort of patients antibody mediated enhancement does not seem to have occurred in anyone.

**Moderna vaccine much easier to store and distribute**

This Moderna vaccine requires cold storage, but only at -4 degrees Fahrenheit, not the -94 degrees Fahrenheit required by the BioNTech/Pfizer vaccine. This can be achieved in existing freezers, and makes dispersement of the vaccine much more logistically feasible.

**Seems likely that Moderna>>BioNTech/Pfizer, but multiple vaccines are needed**

We will need multiple vaccines to beat back this pandemic, but it seems like the Moderna vaccine at this point seems to be preferable to the BioNTech/Pfizer, showing an estimated 50% better efficacy, reporting efficacy vs. severe disease, with substantially more manageable dispersement logistics.

**Some key caveats**

This is GREAT news, and with last week's gives me great hope that we will be able to get this virus under control and end the pandemic via vaccines. There are still caveats -- the comparative placebo vs. vaccine analysis of side effects need to be completed to confirm the vaccines are not unsafe, and future studies will have to confirm that the vaccine is in fact preventing infections, and not just preventing symptoms, which is important given how SARS-CoV-2 largely spreads from asymptomatic infected. It is likely that the vaccines DO prevention infections and infectiousness, not just symptoms, but given the design of these trials only indicated SARS-CoV-2 testing for those with symptoms, we will not be able to confirm this until we see more data.

**The light at the end of the pandemic tunnel coming into view?**

This news reinforces the timeline I have been suggesting this fall for the end of the pandemic -- it is likely that this fall and winter will be brutal with surging cases, hospitalizations, and deaths, but that starting in the Spring vaccine dispersement will start to cut into the spread along with whatever mitigation strategies we are following, and that by summer 2021, the pandemic per se may be effectively over, and we may have a chance to return MOSTLY to normal by fall 2021. That may sound like a long time, but had these vaccines not looked promising, we would be looking at an even more extended pandemic and much more uncertainty.

To me, the light at the end of the tunnel is starting to come into view, so we all need to hang in there so we can arrive safely there together.

Image from __SkyNews__

Obviously the vanishingly small p values are not sufficient for characterizing the efficacy as I elaborate and the confidence intervals are a better sense of precision and effect size. But they are legitimate. The pvalue as are so small because they are testing vs null of no efficacy at all which is a really low bar clearly but I included them to all their digits to make the point that it is CLEAR even from what seems to people not familiar with these designs as a small sample size that the vaccine is preventing cases, at least as defined in the protocol. That is not the main point of the post sorry it was so distracting.

I am wondering if there is enough information such as what the rates for symptomatic cases in various locations are ,when and where subjects were enrolled etc. in order to estimate the expected number of symptomatic cases in the the placebo arm and see if this matches the observed. This would help in deciding if the study results are reasonable.

This is indeed great news. But surely you realize those p-value thresholds are nonsensical. Also, the magnitude of the logs of tiny p-values are not any indication of amounts of evidence. I doubt that p-values are even a good way of judging efficacy here. I, for example, am very suspicious of using a Binomial to interpret these results. At such tiny populations, there's bound to be misspecification which makes these extreme statements of evidence laughable.

Lol. I know it is over the top, but I felt the need to communicate the overwhelming level of statistical evidence given I have seen many misunderstand the design and suggest “sample of 95 not near enough for statistical significance” and p~0.001 not strong enough to convey the level of evidence here. I would of course never write out pvalues like this in paper and never have but am targeting non statistical audience and want them to have no doubt about the overwhelming evidence for this effect

Very important and informative Post, but please avoid to give very small p-values, see.eg. https://errorstatistics.files.wordpress.com/2019/07/nejm-manuscript-statistical-guidelines-2019.pdf

or https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf

p<0.001 is enough.

Helmut Küchenhoff, Statistician, LMU Munich