top of page

Refuting Peter Doshi's claims doubting "trustworthiness & meaningfulness" of COVID vaccine results

Updated: Jan 21, 2021

Peter Doshi has written a piece in the opinion section for the British Medical Journal (BMJ), for which he is an associate editor, questioning the "trustworthiness and meaningfulness" of the published study results for the Pfizer/BioNTech vaccine. In it, he implies a lack of transparency in the analyses and suggesting the vaccine efficacy would be more like 19%-29% based on something he argues may be a "more meaningful clinical endpoint" than the primary endpoint corresponding to the 95% efficacy estimate reported in the FDA report and New England Journal of Medicine paper. His report implies that the pharmaceutical companies are misrepresenting the results in a way that exaggerates their efficacy and safety, and that this misrepresentation has not been caught by the FDA or NEJM peer review.


His opinion piece is the most highly read at BMJ, a highly respected medical journal, and is also being highly cited by anti-vaccine groups all over the internet. If his claims are accurate, then this could be a great service to science to raise legitimate questions and correct the public's perception of the demonstrated efficacy of the Pfizer and Moderna vaccines. But if his claims are not accurate, this piece does a great disservice to the scientific community and world as we struggle to get control of this pandemic.


The Pfizer/BioNTech phase 3 study appears to be well-designed, being a double-blind randomized placebo-controlled trial with clearly pre-specified dose, schedule, and endpoint, and its results appear to be better than anyone dared hope, with the 95% efficacy far exceed the 50% threshold for FDA approval and an adverse event profile that appeared positive, with transient fever, fatigue, headache, and arm soreness among the few that differ from placebo. While there are questions remaining about durability of immunity, degree of protection against asymptomatic disease or transmission, and long term safety endpoints, these results surpassed expectations and appeared extremely promising. Is it possible that these results are not what they appear? Let's investigate Doshi's claims.


FDA report reveals that not all COVID-19 cases counted in primary efficacy analysis?

The crux of his argument is based on a section in the FDA report entitled "Suspected COVID-19 Cases" that he portrays as evidence that the vast majority of true COVID-19 cases were omitted from the primary analysis to inflate efficacy estimates, and that if all true COVID-19 cases were included the efficacy would be much lower. He presents this as some sort of "gotcha" moment, but looking at the documents, it is clear he is either misunderstanding or misrepresenting the trial and its results.


First, let's see this section in the FDA report upon which he builds his claims, which consists of two paragraphs at the top of page 42. The first paragraph starts with the sentence "As specified in the protocol, suspected cases of symptomatic COVID-19 that were not PCR- confirmed were not recorded as adverse events unless they met regulatory criteria for seriousness," and then goes into the details of the two suspected cases in the trial that met the regulatory criteria for seriousness.


The second paragraph goes on to discuss the total number of "suspected cases of symptomatic COVID-19", and since it is the springboard for Doshi's argument, here I will copy the entire paragraph:


"Among 3410 total cases of suspected but unconfirmed COVID-19 in the overall study population, 1594 occurred in the vaccine group vs. 1816 in the placebo group. Suspected COVID-19 cases that occurred within 7 days after any vaccination were 409 in the vaccine group vs. 287 in the placebo group. It is possible that the imbalance in suspected COVID-19 cases occurring in the 7 days postvaccination represents vaccine reactogenicity with symptoms that overlap with those of COVID-19. Overall though, these data do not raise a concern that protocol-specified reporting of suspected, but unconfirmed COVID-19 cases could have masked clinically significant adverse events that would not have otherwise been detected."


Doshi argues that these 3410 cases of "suspected but unconfirmed COVID-19" should have been included in the primary efficacy analysis, suggesting a better choice for the efficacy end point may have been a comparison of the total confirmed + suspected COVID-19 cases between the vaccine and placebo arms, which would have led to much lower efficacy estimates.


The 95% efficacy measure came from the fact that of all the confirmed symptomatic COVID-19 cases in the study, 162 were in the placebo arm and just 8 in the vaccine arm, suggesting (162-8)/162 = 95% of the cases that would have occurred sans vaccine were prevented by the vaccine. He argues that these "suspected but unconfirmed" cases should also be included, which would have 1594+8=1602 cases in the vaccine group and 1816+162=1978 in the placebo arm, which would result in an efficacy of (1978-1602)/1978=19% efficacy, far below the FDA bar, or if leaving out the 409 in the vaccine group and 287 in placebo group that were within 1 week of the shot and might be side effects of the vaccine, then it would be 1602-409=1193 cases in vaccine arm and 1978-287=1691 cases in the placebo arm, for an efficacy of (1691-1193)/1691=29% efficacy, still below the FDA bar.


He puts these forward as more valid efficacy measures and then mentions "Pfizer’s 92-page report didn’t mention the 3410 “suspected covid-19” cases. Nor did its publication in the New England Journal of Medicine. Nor did any of the reports on Moderna’s vaccine", implying these numbers were purposefully left out of all reports perhaps with malicious intent, presenting this as a "gotcha" result and implying the exceptionally strong 95% efficacy results for Pfizer (and Moderna) are a mirage created by suppressing legitimate COVID-19 cases counts.


Who are these "suspected but unconfirmed COVID-19 cases," and why should they be included in efficacy analyses?

If you look through the protocol, which was posted in its entirety months ago, you see no mention of a category of COVID-19 cases that are "suspected but unconfirmed." The only time "suspected" is used in the protocol is in the context of determining when a "potential COVID-illness visit" should be scheduled to give a SARS-CoV-2 PCR test based on reported symptoms. Since it was deemed infeasible to perform weekly PCR COVID-19 tests for all >30k subjects to identify asymptomatic disease, this trial (as well as the Moderna one) focused on demonstrating efficacy vs. symptomatic disease. As described in Section 8.13, page 88/137 of the protocol, they did this by instructing subjects to immediately contact the site for a SARS-CoV-2 PCR test if they experienced any of the following symptoms anew that COULD indicate potential COVID-19: fever, cough, shortness of breath, chills, muscle pain, sore throat, loss of taste/smell, diarrhea, and vomiting. The protocol states that anyone reporting one of these symptoms (except within 7 days of receiving shot) is a suspected COVID-19 patient who should immediately arrange a potential COVID-19 illness visit to receive a PCR test to determine if they are infected with the SARS-CoV-2 virus or not. Those who receive a positive PCR test are "confirmed COVID-19 cases" that are included in the primary efficacy analysis, and those whose PCR test is negative (or if done multiple times, repeatedly negative) are "suspected but unconfirmed COVID-19 cases."


Thus, it is clear that "suspected but unconfirmed COVID-19 cases" are all of those who reported one of those flu-like symptoms but whose SARS-CoV-2 PCR test came back negative.


The only reason why it might be appropriate to include these in the efficacy endpoint are if one believes that most or all of them are false negative test results, which Doshi implies is a possibility given his statement "If many or most of these suspected cases were in people who had a false negative PCR test result, this would dramatically decrease vaccine efficacy." However, it is completely implausible that many or most of these 3410 were COVID-19 cases with false negative SARS-CoV-2 PCR tests, given that for this to be the case it would have to be true that:

  • 3410/~30,000 = ~11.4% of study participants were infected with symptomatic COVID-19 within ~2 months during a time in which the confirmed case rate was ~0.5%/month in USA.

  • Since "suspected case" implies symptoms were reported, this would not even include asymptomatic cases, which are estimated to be 50-80% of all cases, suggesting that the true infection rate in the population of trial subjects would then be ~20-50%, which is clearly implausible.

  • the false negative rate would have to be FAR beyond any documented or reasonably posited levels. The nominal false negative rate for the PCR test mentioned in the Pfizer protocol is 2% (see table 3 of FDA report for this PCR test), which would suggest that 3/3410 of these suspected cases were in fact false negative COVID-19 cases, and the updated efficacy would be (174-10)/174 x 100 =93.9% if we pessimistically assumed the vaccine efficacy was 0% for all false negative cases. Even if we assumed the FNR was 10x higher than reported, say 20%, this would mean we expected 43/3410 of these suspected cases to in fact be false negative COVID-19 cases, which would correspond to an updated efficacy of (183-29)/183 x 100 =84.2% if were pessimistically assumed the vaccine efficacy was 0% for all false negative cases. Even assuming a high 20% FNR, this would only result in ~1% of the 3410 being false negatives, obviously far short of "many or most".

  • The high false negative rates that would have to be assumed to propose "many or most of the 3410" are false negatives would imply that essentially everyone in the USA has already been infected with SARS-CoV-2 given the >24 million confirmed cases in the USA for which the individual received a PCR test and a positive test result was obtained.

  • To expect "many or most" of these to be false positive COVID-19 cases also implies that there were very few cases of common cold, seasonal allergies, or food-based nausea or diarrhea in this cohort of ~30,000 patients for the study duration that may have produced any of those common flu-like symptoms outside of COVID-19.

It is obvious that to suggest that "many or most" of these are false negative COVID-19 cases is patently absurd.


He seems to acknowledge the possibility that at least some of these might not be false-negative COVID-19 patients, saying "some or many of the suspected covid-19 cases may be due to a different causative agent", yet still argues that they be considered in analyses because "Why should etiology matter?" The argument he makes on this basis is even more questionable, since it implies that the efficacy of a SARS-CoV-2 vaccine should be evaluated on its ability to prevent all flu-like viral illnesses and their symptoms, which is not a reasonable expectation.


Thus, the crux of Doshi's article claiming that these "suspected but unconfirmed cases" should have been included in the efficacy analysis is based on specious arguments and demonstrates a misunderstanding or misrepresentation of the details of the trial and their data.


He also raises some other concerns about 371 subjects left out for protocol violations, NSAIDs use, and speculates that the NSAIDs use or off-protocol serology antibody tests could have led to unblinding. These are minor concerns in his article relative to the purported suppression of "suspected but unconfirmed cases," and he does not detail how these standard practices would invalidate the strong efficacy results in the published paper and FDA reports, so I do not further address them here.


He finishes with a call for sharing the raw, patient level data for the trial, and chastises companies for not doing so yet. I can appreciate a call for transparency and data sharing, but he does not make it clear what exact data he is seeking, what analyses he is proposing, or what he expects such analyses to reveal. As all biomedical researchers understand, it is difficult and often impossible to share raw, patient-level data because of privacy laws and the possibility of ascertaining the identity of trial participants, and these trials have already been marked with unprecedented transparency with full confidential protocols publicly available since trial inception, the public posting of full FDA reports containing detailed summaries of all primary and secondary efficacy and safety endpoints even before the meeting, and the prompt publication of results in a leading peer-reviewed journal.


I am in favor of full transparency and open scientific critique, and appreciate genuine attempts to challenge weaknesses in clinical trial or regulatory procedures, or raise questions about important factors that may have been missed in published reports. However, it is important for someone, especially someone in a position of influence like associate editor of a major medical journal, to get their facts right before actively discrediting published studies, and especially when the net effect will obviously be sowing distrust for a vaccine during a global pandemic.


I find the substance of Doshi's criticisms to be sorely lacking, and think this opinion piece does far more harm than good in our societal efforts to understand the true efficacy and safety of these vaccines and utilize them as appropriate to combat the pandemic.


Peter Doshi is an Associate Professor of Pharmaceutical Health Policy Research at the University of Maryland School of Pharmacy. He earned an A.B. in anthropology from Brown University, an A.M. in East Asian Studies from Harvard University and Ph.D. in history, anthropology, and science, technology and society from MIT, and his research involves critical assessment of the pharmaceutical industry and FDA procedures, including arguing for greater transparency and data sharing. He is an outspoken critic of Influenza vaccines, and has written numerous commentaries throughout the COVID-19 pandemic critical of the vaccine study designs and results, and their emergency use authorization by the FDA.

28,076 views25 comments

25 Comments


Jeffrey Morris asserts that he has "no affiliation with pharma and no conflict of interest."


I note that he is "a professor and Director of Biostatistics at the Perelman School of Medicine at the University of Pennsylvania."


Apparently, a company called Verismo Therapeutics sponsors research funding at the Perelman School, and I expect that this is not the sole source of funding at the School.


I'm sure readers would be grateful to Professor Morris if he would clarify the following:


1. Which other biotech / pharma companies fund his employer?

2. What are the potential implications for that funding and his employment at the School if he published views contrary to the commercial interests of those companies?

Many thanks.

Like
Jeffrey Morris
Jeffrey Morris
Jul 20, 2022
Replying to

I have no affiliation from Pharma, no consulting or research funding from Pharma, and no research funding for this work I am doing on my blog. Not one cent. I am doing it in my spare time outside of my normal research, teaching and administrative responsibilities, and have no conflict of interest. In fact, it costs me $$ to do this blog as I pay for all of the web fees out of pocket, and the time I voluntarily spend on this is time I could be spending pushing my research career forward or doing consulting from which I could very easily earn more money. But I choose not to because my intent in doing this has nothing…


Like

Just wanted to point out that the link to the FDA report in the first paragraph takes you to FDA's Moderna report, not the one from Pfizer (which I guess it was meant to?)- Also, the link to the entire protocol a bit further down is not working

Like
Jeffrey Morris
Jeffrey Morris
Jan 25, 2022
Replying to

Thanks — I’ll try to fix the links — I found the url for the Pfizer protocol was changed — I downloaded so can post the actual pdf in the post

Like

We would have detected the additional 6 deaths in the raw data. Now we know Doshi was CORRECT! https://www.thegatewaypundit.com/2021/11/huge-breaking-news-pfizer-hid-public-number-deaths-covid-clinical-trials-actual-number-21-6-reported-4-unvaccinated-group/

Like
Jeffrey Morris
Jeffrey Morris
Nov 17, 2021
Replying to

Your article is inaccurate At the end of the blinded randomized period there were 14 and 15 deaths in vaccine and placebo groups, essentially equivalent. And then many of the placebo crossed over and were offered vaccine and the 6 Additional deaths, while in people vaccinated, were split even between the randomized placebo and vaccinated group so are not linked to vaccination. So there is no evidence whatsoever of greater risk of death in vaccine than placebo from the final clinical trial results. And on relating this point to Doshi — Doshi’s points in the referenced commentary had nothing to do with this, and were erroneous for the exact reasons I lay out in this blog post

Like

Boris Strunjaš
Boris Strunjaš
Nov 15, 2021
Like

The Scrutinizer
The Scrutinizer
Mar 17, 2021

3410 is a significant number of clinical trial participants to be placed in an ‘iffy’ category (suspected but unconfirmed), what does that even mean? On one hand we have sophisticated scientific methods like producing an mRNA vax, accuracy to a single precision (i.e. 7 decimal places), on the other there is question of an invisible little number like 3410 participants in a clinical trial no less….?

The question of ‘False -ves with RTPCR’. If the people setting up the test lack the understanding of how the predictive value of RTPCR for COVID-19 critically depends on the time from Exposure and Symptom onset, then YES there would be false negatives. The false-neg rate is lowest 3 days after onset of symptoms,…

Like
The Scrutinizer
The Scrutinizer
Mar 18, 2021
Replying to

I'm looking at the data as a scientist, a vaccinologist, to get answers from the trial results and the setup as well. This is not called being skeptical, it's called making sense of the incongruous protocols and results. You are extrapolating, that is NOT called science.

"But now since the phase 3 trial...." were the asymptomatic cases higher in the controls then? Also, if in these studies you mention, infections were greatly (??) reduced, why compare with asymptomatic people. doesn't really make sense (problems with rationalizing a result rather than supporting it with mechanism). Anyways, I would love to take a look at these studies that you mentioned, would you happen to have the reference?

Lastly to address your comment…

Like
Post: Blog2_Post
bottom of page