To what extent is societal vaccination saving lives? Are the vaccines really safe?
Some active vaccine skeptics will present bits of data suggesting the vaccines "aren't working" or use open reporting systems like the USA's VAERs to claim that the vaccines are causing many deaths, some even suggesting these deaths outweigh any life-saving benefit of the vaccines in preventing COVID-19 deaths.
Many of these arguments are based on simplistic arguments, like plotting the COVID-19 death or infection rate for various countries in the world vs. the overall vaccination rate as in this paper, or by highlighting anecdotal cases in which countries with few cases or deaths in the past year have their first real COVID-19 surge this summer, after vaccination is already underway.
These analyses are simplistic and fundamentally flawed for many reasons, and tend to lead to inaccurate and misleading conclusions. Many factors affect the viral dynamics in a given time and place besides vaccination rate, including the rate of previous infections, seasonal effects affecting how much time people spend indoors, the degree to which people are exposed to others based on population density and local mitigation policies in place, the types of vaccines used and how long since they've been given, the age distribution of the vaccinated/unvaccinated cohorts, and the current societal infection rate that determines probability of exposure.
These factors introduce confounding of time and age effects into the raw data, and it takes rich data including measurements over time, split out by vaccine status and age along with knowledge of vaccination rates in the various age groups over time, in order to properly adjust for these and hope to gain valid insights into the real-world effect of the vaccines. As shown previously, failure to adjust for age effects or time effects and computing overall summaries of severe disease or death will tend to produce distorted summaries as a result of Simpson's paradox.
The lack of such data has limited our ability to rigorously assess the real-world effect of vaccines in many places in the world, and has led many to feel the need to resort to simplistic summaries that don't paint an accurate picture of what is going on.
Fortunately, some countries have such data and have made it publicly available. One is the United Kingdom, who is sharing extensive data related to the pandemic on their official government website. Their data has been valuable in understanding some of the emerging dynamics of the pandemic, including vaccine effects, emergence of Alpha and then Delta variants, and the waning of immune protection vs. infection that appears to start 5-6 months post vaccination. In this post, I will focus on a particular data set they posted on November 1 looking at deaths in the UK from January 2, 2021 through September 24, 2021 split out by age and vaccination status. Here is a link to the Excel spreadsheet containing the data.
I focus on deaths, since this is really the key factor when it comes to both effectiveness and safety -- it allows us to assess to what degree do the vaccination programs seem to be reducing deaths due to COVID-19, and also to assess whether there are any safety signals to suggest that the vaccines might be causing an uptick in deaths in some groups, as claimed by some groups of vaccine skeptics. They post both COVID-19 related deaths, defined by deaths with known recently positive SARS-CoV-2 status (COVID-19 deaths), as well as all cause deaths.
I will consider both COVID-19 deaths and all-cause deaths, in a series of 4 blog posts:
Part 1: Comparing COVID-19 deaths across unvaccinated and vaccinated (split out by 21 days since 1st dose, >21 days after 1st dose, and after 2nd dose)
Part 2: Comparing all-cause deaths between unvaccinated and fully vaccinated (2 doses)
Part 3: Comparing all-cause deaths between unvaccinated and recently vaccinated (<21 days since first dose).
Part 4: Comparing all-cause deaths between unvaccinated and partially vaccinated (>21 days since first dose)
This replaces the blog post I briefly posted on 11/22/21 that was linked to only the COVID-19 deaths, not all cause deaths. The first blog post presents and discusses these results, while the two that follow will delve into all-cause deaths, first 2-dose and then 1-dose.
This blog post will consider COVID-19 deaths, since these are considered the primary clinical efficacy signal of the vaccines -- preventing deaths due to COVID-19 after SARS-CoV-2 infections. In these data, a death is considered a COVID-19 death if either of the ICD10 codes U07.1 (COVID-19, virus identified) or U07.2 (COVID-19, virus not identified) is mentioned on the death certificate.
Because death attribution is imperfect, and because we are also interesting in considering any deaths potentially related to vaccination, in the subsequent 2 posts we will analyze all-cause deaths. Some caveats re: denominators of unvaccinated death rates for UK Data:
There has been some confusion resulting from the publicly posted UK Data regarding SARS-CoV-2 vaccinations because of uncertainty in estimating the number of unvaccinated UK individuals in each age group that are used in the denominators for any calculation of death (or infection) rates for unvaccinated UK residents can lead to potentially severely biased estimates.
The problem is that there is no way to directly measure number of unvaccinated, but given the number vaccinated and total population estimates, they have to be estimated by subtraction. The problem is that it is surprisingly difficult to get accurate age-specific population estimates, and in age groups with high % vaccinated (like the older groups), even a small increase in population estimates can dramatically increase the estimated number of unvaccinated, which in turn dramatically decreases any unvaccinated death rate estimates.
This problem is detailed in the following document. Briefly, there are two different sources of population:
Office for National Statistics (ONS) 2019 mid-year population estimates: These are the most recent and considered the best publicly available population estimates. They are based on projecting from the 2011 Census, so are subject to uncertainty and may be an underestimate.
National Immunisation Management System (NIMS) denominators: These are the number of individuals registered with the National Health System (NHS) who are currently alive in the resident population. These likely overestimate the population, given that there are concerns that they include people who no longer reside in the UK,
John Burn-Murdoch (@jburnmurdoch), Chief Data Writer for the London Financial Times, describes this phenomenon in this article here, He explains how the well-publicized phenomenon based on data from the UK Health Security Association (@UKHSA) using NIMS denominators and showing the infection rates more than doubled in vaccinated than unvaccinated individuals, a result not observed other places. If ONS denominators were used, the number unvaccinated in various age groups is less than half as much (e.g. For 50-59yr, 400k vs. 1m), more than doubling the unvaccinated rates, and changing the conclusions significantly to find the vaccinated individuals had slightly lower infection rates.
Since the present report upon which this blog post is based, I assume the ONS denominators are being used. However, since this is not stated explicitly in the definitions page of the spreadsheet, and this caveat of which denominators are used is important to consider when trying to interpret any public UK data involving death (or infection) rates for unvaccinated individuals.
Summary of Data
Table 3 in the spreadsheet contains weekly data from the UK including COVID-19 death counts split out by age group (10-59, 60-69, 70-79, 80+) and vaccination status (unvaccinated, 1st dose within past 21 days, 1st dose >21 days ago, 2nd dose). They also have the population number for each vaccinated status from each age group for each week that can be used to compute standardized death rates as well as to track the proportion of that age group in each vaccination status over time.
This is one of the few large-scale data sets I've seen in which death is reported over time, split out by vaccination status and age, the key factors that MUST be accounted for in order to begin to make any valid assessment of the real-world effect on vaccinations.
It is possible to fit rigorous statistical models to disentangle all of the confounding factors of vaccination rate, age, time, and vaccination status to tease out valid real-world effectiveness measurements, and some excellent papers have done so. However, to keep things simple and accessible, I will not do that here. Instead, I will simply plot the all-cause death rates over time for each age group and vaccination status. If done carefully, these plots can parse out the confounding effects and provide a clear visual picture of the effect of vaccination on all-cause death in the UK in 2021. This data set runs through late September, allowing enough time to accurately assemble the death data for the November 1st release.
To make the plots maximally informative, the data are plotted as follows:
I will plot the COVID-19 death rates, which are the total COVID-19 deaths in that week for the given age/vaccination status group divided by the population of that group during that week. Plotting the raw death numbers would be misleading since it would not adjust for the number vaccinated, a measure varying substantially over time and by age group. Using these death rates automatically adjust for this factor.
I will plot separately by age groups. Since both risk of death and vaccination rate vary strongly across age groups, failure to stratify by age under these circumstances can produce extremely distorted and misleading results as a result of Simpson's paradox, as previously shown in Israeli data. Stratification by time avoids this effect.
Further, we will plot the entire time curve, not overall summaries. Key pandemic factors including the overall SARS-CoV-2 infection levels, the predominant variant, the season of the year, and the mitigation strategies in place all change over time and could have a strong impact on death rates, and of course the vaccination rate has also changed substantially over time. Summaries that aggregate over time under these conditions can also produce a distorted and misleading picture of reality via a Simpson's paradox effect, as shown in a previous blog post. Plotting the entire time curves avoids this effect.
The population size of each age/vaccine status group changes greatly over time, with the vast majority of the population being unvaccinated at the beginning of the year, and becoming increasingly vaccinated as the year progresses, with the older groups vaccinated earlier and more extensively. To illustrate this effect, I will make the thickness of the line for each time plot proportional to the % of that age group with that vaccination status. By doing this, we can account for the population size for each vaccination status/age group over time, and can also see for a given age group when most received their vaccination, as indicated by the time of the year in which the thickness of the vaccinated lines is increasing rapidly.
For maximal transparency, here I share the script I used to download the data and produce these plots, so anyone can see the data for themselves and adapt the code to change the figures in any way they want. The plotting is done in the freely available statistical package R,
Here is the plot for the oldest group, that is 80+ years old:
From this, the dominant feature is the huge spike of excess deaths in the winter that resulted from the massive Alpha COVID-19 surge, that impacted primarily the unvaccinated population. Some other observations:
Since vaccination started with this age group, many in this age group received their first dose in early January, and second dose by February to March, as indicated by the thickening of the lines for 1st dose and 2nd dose vaccinated and thinning of the line for unvaccinated during those times.
The UK has experienced a massive Delta surge in the summer, and note the lack of high death rate from this surge, even in the unvaccinated group. The death rate is much lower than in the Alpha surge in the winter.
Even so, the COVID-19 death rate is considerably lower in the vaccinated group. For example, in week 35 for the 96% in this age group who are vaccinated with 2 doses, the death rate is 97 per 1 million, while for the 2.8% unvaccinated, it is 488 per 1 million, more than 5x higher. So the rate of all cause death in fully vaccinated is 5x lower than unvaccinated.
Here is the data for the 70-79yr old group:
In this age group, we see the same massive surge of deaths in the winter Alpha surge, again focused primarily in the unvaccinated population. Some other observations:
We see the vaccination started slightly later in this age group, with first doses given a few weeks later in January, and second dose rates increasing in March.
Again, we see the Delta surge in the summer produced far fewer deaths than the winter Alpha surge, even in the unvaccinated.
Even though the vast majority of this age group is vaccinated with 2 doses (96.2% by week 35), there are still 3% unvaccinated and ~1% who have only received 1 dose.
The COVID-19 death rates during the Delta surge are higher in the unvaccinated (328 per 1 million in the 3% of the population still unvaccinated in week 35) and the small number who have not received a second dose (299 per 1 million in the 0.7% of the population >21 days after 1st dose) than the fully vaccinated group (33 per 1 million in the 96.2% of the population vaccinated with two doses), with the fully vaccinated showing a 10x reduction in overall death rate in week 35.
We see a few spikes in the purple plots in weeks 15, 19, and 22 suggesting higher death rates in some individuals within 21 days of first dose. Before the vaccine skeptics claim "Aha, here is the evidence of vaccines killing people", note the small sample sizes -- In week 15 the spike corresponds to 2 deaths out of 11,503 within 21 days of first dose, the week 19 spike is 1 death out of 4427, and the week 22 spike is 1 death out of 4175. In the entire country. So while it is possible that the vaccine may (or may not) have played a role in these 4 deaths, it is certainly not evidence of some sense of danger in the vaccines.
Here is the data for the 60-69yr old group:
For this age group, we see the winter Alpha surge produced an uptick of deaths, but nearly as dramatic as we saw in the 70-79 and 80+ age groups. But once again, the death rates were higher in the unvaccinated than vaccinated cohorts during this time. Some other observations:
We can see this group primarily received their first doses in February, and their second doses at the end of March and into April.
We can also see the summer Delta surge in this age group. Note that the COVID-19 death rate in the unvaccinated during the Delta surge was of the same order of magnitude as the Alpha surge in the wintertime, unlike the 70-79 and 80+ age groups in which the Delta surge death rates were MUCH lower than the Alpha surge death rates. Although note the Delta summer surge death rate in 60-69 is ~100 deaths per million, still much less than the ~200 and ~600 death per million seen in the 70-79 and 80+ groups.
By summertime, the vast majority of this cohort have been fully vaccinated, with 93.7% given 2 doses, 1.2% 1 dose >21 days ago, 0.1% with 1 dose given <21 days ago, and only 5.1% unvaccinated.
The COVID-19 death rates in the summer Delta surge are much higher for the unvaccinated population (128 per 1m, 5.1% of population) and the single vaccinated group (105 per 1m, 1.2% of population) than the fully vaccinated (11.7 per 1m, 93.7% of population), with the fully vaccinated showing all cause death rate >10x lower than the unvaccinated.
Again we see three large purple spikes in week 29, 32, and 33 showing spikes in "death rate" for those receiving 1st dose within the previous 21 days. This is from a very small sample group, indicating a single death among 6250 within 21 days of first dose in week 29, out of 4236 within 21 days of first dose in week 32, and among 3874 within 21 days of first dose in week 33. It is not clear whether these 3 deaths in the UK were related to vaccine or not, but again, there is NO evidence of the vaccines being deadly dangerous here.
Finally, here is the youngest cohort 10-59.
As will be pointed out in part 2 of this 3-part blog series, these UK data have major problems for the 10-59yr old age group, since this age group is too large, with the older end of this age interval having people with much higher vaccination rates and much higher risks of death than those at the younger end. This confounding leads to a Simpson's paradox artifact in which the vaccinated COVID-19 death rates will be inflated relative to the unvaccinated COVID-19 death rate. They really should have split out the 10-59yr into finer age groups with more homogeneous vaccination rates and death risks -- too bad they didn't.
This caveat notwithstanding, we will nonetheless discuss these results.
For this age group, we also see an uptick in all-cause deaths from the winter Alpha surge, but note that the COVID-19 death rate is much lower than the other age groups, with a peak death rate near 15 per million in the unvaccinated relative to 150 per million in the 60-69, 1250 per million in the 70-79, and 9000 per million in the 80+ populations. Again, the unvaccinated have higher COVID-19 death rates during this surge, although there were very few vaccinated (only 4.3% within 21 days of first dose, 0.7% >21 days since first dose, and 0.3% given second dose by week 3). Some other observations:
We can see that vaccination didn't ramp up in this group until March and April, with 2nd doses not coming until April too May.
Even by summertime, there remained a large proportion of unvaccinated individuals in this cohort. By week 35, only 63.4% had received 2 doses, 7.2% >21 days since 1st dose, 2.0% <21 days since first dose and a full 27.4% unvaccinated still as of August.
During the summer Delta surge, we see another uptick in COVID-19 deaths in this age group, and once again the COVID-19 death rates are higher in the unvaccinated (5.7 per 1 million, 27.4% of population) and single dose (5.5 per 1 million for <21 days since first dose, 5.6 per 1 million for >21 days since first dose) groups than the 2 dose vaccinated group (2.5 per 1 million, 63.4% of population). While the COVID-19 death rates are much lower than the other age groups, and the magnitude of reduction of COVID-19 death rate is smaller, we still see the fully vaccinated individuals in this age group have 1/2 the risk of COVID-19 death than the unvaccinated/partially vaccinated cohort.
We see 2 purple spikes apparently indicating increased COVID-19 death rates in those <21 days since first dose, but again these are small sample size artifacts, with 2 deaths among 481,233 given first dose within 21 days of week 30, and 1 death among 231,831 individuals receiving first dose within 21 days of week 38.
Conclusions:
In this blog post, I have plotted and interpreted COVID-19 death rates over time split out by vaccination status for the various age groups. The carefully chosen plotting techniques enable parsing out some of the confounding effects of time, age, vaccination rate, infection rate, and variant and get a fair assessment of how the UK vaccination program is affecting COVID-19 deaths.
In all age groups, we clearly see the vaccinated groups having lower risk of COVID-19 death, and this is especially evidence during the winter Alpha surge and summer Delta surges. The vaccinated individuals appear to be MUCH less likely to die in a COVID-19 surge, with fully vaccinated individuals in week 35 having 5x, 10x, 10x, and 2x lower all cause death rates in the than respective age cohorts than unvaccinated individuals. This is consistent with the vaccines protecting strongly against death even after some waning of circulating antibodies as has been noted at 5-6m post-vaccination.
This is particularly impressive given that we know 1/2 of the UK population has received AstraZeneca vaccines, that supposedly are less efficacious than the mRNA vaccines. It is possible they are similarly effective in preventing death even if they are slightly less effective in protecting vs. infection. It would be great to see updated data from October and November, but given the time lag to confirm deaths, late September is the most recent they could release in early November. I look forward to seeing updated data from the UK, and using the script I share above, it is easy to immediately produce these plots again when the updated data are released. Additionally, the scripts can be used to produce similar plots from data from other countries should data on all cause deaths plot out by vaccination status and age over time are released, as I hope they are. It can't be emphasized enough how much better these data are than the naive scatterplots of national infection or death rates vs vaccination rate at an arbitrary point as published in this (weak) paper by Harvard researchers that has become a favorite of vaccine skeptics, or discussions of the raw infection or death rates over time, not split out by vaccine status or age, for hand-picked countries that seem to fit a narrative of "why are infections going up after vaccination if vaccines work?" As scientists know, accurate causal interpretation of observational data is tricky, especially in the setting of the pandemic with vaccine rates and event risks wildly varying across age groups and over time and place as surges of different variants hit. As a result, careful analysis is more crucial than ever. Hopefully, the increasingly availability of thorough data like the UK data analyzed here make it possible for us to clearly understand important factors like how vaccination is impacting the population. This blog post focused on COVID-19 death rates, but this depends on death attribution practices in terms of COVID-19 being listed on the death certificate, which also may be partially dependent on SARS-CoV-2 PCR testing practices and thresholds that may have changed over time. As a result, it is important to look at the all-cause deaths. Fortunately, the UK ONS has shared these data as well, presented weekly and split out by age and vaccination groups, so we can look carefully at these to see what information they provide us on the real-world impact of the UK vaccination program on all-cause deaths.
Part 2,Part 3, and Part 4 of this blog series compare all-cause deaths for unvaccinated and fully vaccinated (2-dose), recently vaccinated (<21 days after 1st dose) and partially vaccinated (>21 days after 1st dose), respectively.
This will enable us to assess whether there is a "safety signal" for death after vaccination in a manner that is completely independent of any SARS-CoV-2 testing or COVID-19 death attribution practices.