OK -- one more Simpson's paradox themed post. There is one more major point I want to make sure people realize to avoid misinterpreting some of the commonly used simple summaries of infection, hospitalization and death rates with respect to vaccination that people look at to get a "common sense" indication of how well "vaccines are working"
Three previous examples show how age confounding can make vaccines look worse than they are, decreasing estimates of vaccine effective (VE) if computed overall, i.e. not stratifying by age or adjusting for age effects in the model in some other appropriate way. We showed this in 3 settings:
All three examples had the same paradoxical results whereby overall VE estimates were MUCH lower than any of the age-specific VE estimates when broken down by age. These were all illustrations of Simpson's paradox, caused by the fact that the confounder (age) was strongly positively associated with the exposure (vaccination) and outcome (serious disease/death). However, Simpson's paradox works both ways -- it is also possible for an unadjusted confounder to make vaccines look better than they are. If the confounder is strongly positively associated with the exposure but strongly negatively associated with the outcome, then Simpson's paradox can make the overall VE appear much higher than if computed separately for different levels of the confounder, and thus provide a misleading account of vaccine effectiveness.
I have suspected this phenomenon is responsible for inflating some unusually high estimates of vaccine effectiveness in the USA in the spring and early summer, with numbers >96% reported, creating the misimpression that vaccines were preventing nearly all COVID-19 infections, hospitalizations and deaths, raising expectations unrealistically high for the ability of vaccines to completely protect vs. COVID-19. In this blog post, I will illustrate how, with the dynamics of vaccination, infections, and COVID-19 deaths over time, time confounding can create a Simpson's effect that artificially inflates VE estimates whenever they are computed over a very long time interval (e.g. all of 2021) and the analysis is not stratified by time or otherwise adjusted for time effects in the modeling. Here is a summary of the factors leading to this effect:
Vaccination rates were very low in early 2021
COVID-19 infection/death rates were very high in early 2021 from winter surge
Vaccination rates strongly increased moving from winter into spring/early summer 2021
COVID infection/death rates decreased moving from winter into spring/early summer coming off the winter surge and into the pre-Delta lull.
From this, we see that a confounder (time) was strongly positively associated with exposure (vaccination) and strongly negatively associated with the outcome (COVID-19 cases/deaths), and this caused a Simpson's effect that made any vaccine effectiveness estimates (naively) computed using total counts for all of 2021 inflated and misleading, much higher than the VE for each time interval if computed separately.
The lesson of this is that we need to be wary of simple summaries from observational data reported in the media and other places as evidence for or against vaccine effectiveness, whether percent of cases/hospitalizations/deaths that are vaccinated, or simple "vaccine effectiveness" calculations computed from overall numbers, without stratifying or otherwise adjusting for key confounding factors such as age and time and other confounding factors.
With observational data, the presence of these confounding factors makes simple summaries sometimes grossly misleading, and that is why advanced statistical modeling is necessary to try to adjust for key confounding factors if one wants a reliable estimate of vaccine effectiveness.
I will illustrate this phenomenon with hypothetical data, using real USA monthly COVID-19 confirmed case and death counts and vaccination rates, but assuming a fixed 90% vaccine effectiveness number in the monthly data, and showing how distorted the VE estimates get when computed from cumulative numbers without adjusting for the time dynamics. Hypothetical #1: VE vs. Infection First, I will look at how this phenomenon could distort overall estimates of vaccine effectiveness vs. infection. First, I downloaded the following monthly data:
Monthly numbers of % of USA population fully vaccinated from CDC site
Monthly numbers of confirmed cases in USA population from worldometers site
For simplicity, here I use an estimate of USA population of 330 million, and following are the summaries of monthly data over time:
For simplicity, I just dichotomized into fully vaccinated and unvaccinated for this illustration.
The plot on the right shows the trends of percent vaccination and COVID-19 cases over time.
Note how vaccination was just starting during the massive winter surge, and then as vaccination ramped up, the case count decreased, with June levels of infections (124.5) 15x lower than the January (1905.6) levels. Of course vaccines likely played a part in these declining case counts over time, but the massive surge in winter 2021 was bound to decline in the spring anyway (as we've seen happen in past surges), so this coincidence of declining cases and increasing vaccination could lead to confounding of time with the vaccine effect. And this distortion can be pretty dramatic. Don't take my word for it. Let me explicitly illustrate it.
Now, I would love it if we had complete USA monthly case data split out by vaccination status, but sadly we don't have that data. So from here forward I will posit a hypothetical of 90% vaccine effectiveness vs. infection in the monthly numbers and show how the distortion can occur. Let me emphasize here: these efficacy numbers are hypothetical and not a statement of the true vaccine effectiveness in the USA -- this is unknown, and of course has varied over time as different variants have emerged and circulating antibodies induced by the vaccine have waned. My goal here is to illustrate the distortion from time confounding, so I choose a simple fixed rate to be able to clearly see the distortion. Assuming VE=90% for each month, we assume that the relative rate of infections per 100k for vaccinated is 10x lower than the relative rate of infections per 100k in the unvaccinated. Placing this assumptions onto the real USA vaccination and total monthly case data would yield the following table:
Again vaccine effectiveness using the monthly numbers is computed based on the usual relative rate reduction formula:
VE = 1 - vaccinated monthly cases per 100k/unvaccinated monthly cases per 100k
See how this hypothetical has 90% VE for every month of 2021.
What if someone computed VE not monthly, but using the cumulative numbers from all of 2021? That is, if they used the total cumulative cases in vaccinated and unvaccinated groups to compute the VE as:
VE = 1 - total 2021 vaccinated cases per 100k/total 2021 unvaccinated cases per 100k
I'm not saying this is advisable -- it is absolutely not -- but I have seen numbers presented in which this is what they have done, so let's see what happens if we do.
If we compute this VE at the end of each month using all cumulative vaccinated and unvaccinated cases to date, and the number vaccinated and unvaccinated at that time to normalize to cases per 100k, then we would have the following results:
Of course, for January, the cumulative and monthly VE are the same so we get 90.0% for both. But look at February. In spite of the fact that the VE for both January and February separately are 90.0%, when computed cumulatively, we get VE=95.6%. We see as we move into the Spring and Summer, the cumulative VE computed this way goes up almost to 97.8%, even though by construction the VE for each month is 90.0%. This 97.8% would be a misleading number to present if wanting to represent the causal effect of vaccination on infection rates.
This is predominately driven by the previously mentioned Simpson's effect -- nearly all of the early cases during the massive winter surge were unvaccinated given that the vaccination program had just started, and by the time vaccination was underway the winter surge was over and community transmission levels were much lower.
The cumulative VE started to decrease in July and August as the Delta surge produced high levels of community transmission that has weakened the association of time with community infection levels, but not enough to eliminate it: 96.0% would still be a misleading representation of vaccine effectiveness in this hypothetical.
I also included a column in this table that summarizes the % of total cases coming from unvaccinated. As I have emphasized in previous blog posts, this can be a terribly misleading measure since it depends so strongly on vaccination rate, but included here because I have seen it used as a summary measure evaluating vaccine performance in media and social media reports. Notice how we can have a vaccine with 90% effectiveness that could still produce >99% of cases being unvaccinated when we include time periods in which few were vaccinated, once again highlighting how misleading this measure can be.
Hypothetical #2: VE vs. COVID-19 Deaths Next, I will look at how this phenomenon could distort overall estimates of vaccine effectiveness vs. COVID-19 deaths, whose numbers I also downloaded from the worldometers site.
Here are the monthly counts for # of COVID-19 deaths in the USA for 2021, normalized per 100k, again assuming a population of 330 million and the actual USA vaccination rates:
Once again, note how the COVID-19 death rate was extremely high during the winter surge, decreasing >10x throughout the spring and early summer before spiking back up in August once the Delta surge kicked in. Once again, we have a decreasing death rate coinciding with an increasing vaccination rate over time that could potentially provide distortion from a Simpson's effect.
Again suppose a hypothetical that the vaccine effectiveness vs. COVID-19 death is 90%, we would have the following monthly numbers:
If we computed the VE cumulatively at the end of each month, here is what we would find:
Once again, we see how inflated the vaccine effectiveness numbers are if computed cumulatively, effectively ignoring the time confounder. In this contrived setting assuming a fixed VE=90% for each month, we can see how the cumulative VE estimates are so much higher, distorted by the time confounding via a Simpson's effect. These numbers would suggest the vaccines are preventing almost 98% of COVID-19 deaths, even though in this hypothetical scenario it is in fact, only 90%. Note also how high the % of COVID-19 deaths from unvaccinated would be under this scenario, once again driven by the fact that very few were vaccinated in the winter when the major winter surge of cases and COVID-19 deaths occurred.
Conclusions: This blog post is an illustration for how time confounding can be a major factor in the pandemic, with the USA dynamics that, if not taken into account, could lead to grossly inflated estimates of vaccine effectiveness. This dynamic is driven by the fact that cases surged in the wintertime when vaccination just started, and decreased over time coming out of the wintertime surge as vaccinations ramped up. This trend changed a bit as the Delta surge hit in late July and August, but the initial effect still skews vaccine effectiveness estimates if they are computed over too long a time period without adjusting for time confounding. I illustrated this effect using the real monthly vaccination and COVID-19 case/death numbers from the USA, but hypothetically assuming a 90% effectiveness of vaccine in preventing cases and deaths for each month. While based on a hypothetical, the use of real time trends of vaccination and case/death counts over time means this type of inflation effect is almost certain to be experienced in the real USA data if such cumulative VE estimates were computed (as I have seen some do in certain locales this year). In this illustration, I demonstrated that if simple estimates of vaccine effectiveness are constructed using cumulative data from all of 2021, this time confounding causes extreme distortion that inflates the estimates of vaccine effectiveness, producing estimates upwards of 98% in early summer 2021 even though by construction the true VE for each month is just 90%. This shows that Simpson's paradox does not just attenuate effects, but can inflate them as well depending on the relationship of the confounder (time here) and the exposure (vaccination) and outcome (cases/deaths). So what is the take home message from this illustration? Several things:
If you see someone presenting numbers like "% of infected/hospitalized/dead from COVID-19 that are vaccinated," recognize this can be a misleading number to summarize vaccine effectiveness since it depends on % vaccinated, and it is even worse when computed over a long period of time like all of 2021.
When you see people present vaccine effectiveness estimates (vs. cases, hospitalizations or deaths) using the simple relative rate reduction formula, always ask what time frame they are using. If they are modeling over a broad time frame like since the beginning of 2021, there is likely a strong time confounding that will make these numbers misleadingly high. And other countries have similar dynamics so this applies to them as well.
Estimates of vaccine efficacy using this simple relative rate reduction formula should be done over a time frame short enough that we don't have strong variation in vaccination rate or in case/death counts within that interval. Computing VE separately for time intervals short enough would not be affected by this time confounding (but of course might be subject to other confounders other than time). Of course since age is also a major confounder, you would want to stratify by both time and age.
Given there are many potential confounders other than age and time that impact both vaccination rate and risk of COVID-19 infection, including essential worker status, co-morbidities, sex, race, location, and other factors, the best approach for estimating vaccine effectiveness from observational data is to use statistical designs and models that use matching, stratification, propensity score weighting and/or covariate adjustment to try to adjust for all key confounders. This is what is done in the best published peer-reviewed papers coming out of Israel, the UK, and from certain places in the USA, and other countries, and include state-of-the-art approaches for the challenging problem of trying to elicit causal effects of vaccination from observational data. Most careful modeling I've seen includes age groups in the modeling to account for age confounding, or does matching by time periods (weeks or months) or includes time period in the modeling to adjust for the time confounding. Given fine enough data, these models can also account for exposure time for each individual in the vaccinated and unvaccinated groups and thus more precisely adjust for number vaccinated and unvaccinated in the population.
Bottom line is that inferring causal factors like vaccine effectiveness from observational data is challenging, as observational data are rife with confounding factors that can make simple summaries and estimates grossly misleading, sometimes sharply attenuating vaccine effectiveness and making the the vaccines look bad (like from age confounding), and sometimes inflating vaccine effectiveness and making the vaccines look better than they are (like time confounding). Thus, when assessing vaccine effectiveness from observational data, we should look to published papers that use rigorous designs and analytical tools that adjust for the key confounders for our key insights, critically evaluating whether it appears their analysis and design adequately adjusted for the key factors. Simple summaries and estimates have some value, but we need to be cautious in not over interpreting them as I have emphasized in these Simpson's paradox themed blog posts. Age and time are two of the key confounders in vaccine effectiveness, and thus it is crucial that the estimates are stratified by age and time and, if not, we should question whether the interpretations being drawn from the estimates are proper.
So I hope these insights arm the public with tools to critically assess what they are seeing and garner a more accurate sense of how vaccination is impacting the pandemic. Here is the Excel spreadsheet I used for these calculations: