The new Omicron variant is far from the only thing changing about the COVID-19 pandemic. Vaccines, vaccine doses, medical advice concerning vaccines, and the language we use to discuss all of it continue to evolve, so YouGov continues to monitor how we ask people about their vaccination status and what results we're getting. We've tested different kinds of questions and found that question language has only minor effects, but we want to get out ahead of all the changes, especially with boosters, so we're moving to a longer series of questions that ask for more specifics.
We also have tracked how our survey estimates compare to government estimates, though much of the difference may result from the challenges facing official efforts in protecting privacy while tracking the same people through multiple vaccine doses.
Beginning in May, the surveys conducted by YouGov for both The Economist and Yahoo News began using a single five-category question to ask Americans about their personal situation regarding vaccination (“Original Single Question” in the following table).
We shifted to this single personal-situation question from earlier approaches to asking about vaccination after conducting a set of survey experiments showing that its results were consistent with reports on vaccinations from the Centers for Disease Control and Prevention (CDC).
Over the summer months, results from both the Economist/YouGov and Yahoo News surveys began to diverge from the CDC estimates, finding a percentage receiving at least one shot that was lower than CDC estimates. That gap continued to widen in August and, by September, our estimates of the proportion receiving at least one shot were falling an average of 9 percentage points below the CDC estimates. Also, for the first time in September, we started seeing a slight gap in the percentage reporting they were fully vaccinated (averaging two percentage points on the Yahoo surveys of U.S. adults, and four percentage points on the Economist surveys of U.S. citizens). Again, CDC estimates were higher. 
We hypothesized that in asking respondents if they had received the necessary injections to be “fully vaccinated against COVID–19,” the original single question might be causing some confusion given the new availability of booster shots. Just over 10 percent, on average, had been choosing the answer, “I’m not sure about getting vaccinated.” Perhaps some meant they were unsure if they were “fully vaccinated” rather than uncertain about whether to get vaccinated.
So, in early October, we conducted another round of survey experiments on an alternative set of vaccine questions (on the Oct. 1-4 Yahoo News survey of U.S. adults and on YouGov Daily surveys of U.S. citizens conducted between September 30 and October 8) which focus on the two-dose vaccination series, asking separate yes/no questions about the first and second doses (labeled “experimental multi-question” in the charts and tables below).
The experimental multi-question series included a probe about the vaccine manufacturer to allow coding to identify respondents who were “fully vaccinated” according to the CDC criteria: either two shots of two-dose vaccines (like Pfizer-BioNtech and Moderna) or a single shot of the Johnson & Johnson vaccine.
Contrary to our hypothesis, the results showed no statistically significant differences in the estimates of those reporting one dose or the percentages fully vaccinated on either survey, both for all adults and for seniors. (The vaccination rates appear slightly higher for seniors on the Yahoo News survey experiment, but these differences are not significant given the smaller sample sizes involved.)
One intriguing aspect of the findings is the way the gap between one dose and fully vaccinated narrowed slightly using the experimental two-dose series questions, a result that moved our survey estimate even further away from the CDC estimates. For example, during the period in question (September 30 to October 3), the CDC estimates show a 13-point gap between those with at least one dose (78% on October 3) and those fully vaccinated (65%). The YouGov Daily experiment, however, showed a five-point gap using the original single question (70% vs 65%) and a narrower, two-point gap (70% vs 68%) with the experimental questions that were theoretically less confusing.
Since the experiments showed no significant overall differences, we opted to continue asking the original single question but also to proceed with further research and experimentation. The recent expansion of booster eligibility to all adults, and the likely coming change in the meaning of “fully vaccinated,” made the need for an alternative approach more urgent. We conducted a second round of experiments using a new set of alternative questions (this time on the Economist/YouGov surveys of U.S. citizens) focusing on the number of shots received in a three-dose series. These new questions (labeled “new multi-question” in the charts and tables below) differed from the prior experiment by asking a follow-up probe of those with at least one shot, asking about the number of shots they have received (one, two, or at least three).
Once again, as with the prior experiment, the new multi-question series produced essentially the same estimates of the proportion receiving at least one vaccine dose. The slight differences – 69% on the original single question and 70% on the new questions – were not large enough to be statistically significant. The same was true for the two-point difference (83% vs 85%) among seniors. Consistent with the previous experiment, the new questions again found slightly smaller percentages classified as receiving one dose but still needing a second to be considered “fully vaccinated” (2% vs 5% for all adults, 1% vs 3% for age 65+). However, this time the new questions also produced a slightly larger percentage classified as “fully vaccinated” (67% among all adults) than the single five-category question (64%). Among seniors, the difference was slightly larger (85% vs 80%).
To sum up: Our question experiments show question wording matters little when it comes to reported vaccination rates, though the term “fully vaccinated” may have become more problematic in recent months. Given the coming legal changes to the meaning of that term, and given our need to track booster shots going forward, YouGov has shifted to using the new question sequence that asks about the number of shots received on surveys conducted for the Economist and Yahoo News.
This leaves open the question of why YouGov’s survey estimates of the percentage with just one shot (70% among all adults during the course of these experiments) continue to lag behind the CDC estimates (an average of 82% over the same period). It is certainly possible that YouGov’s sampling and weighting lead to an understatement of those with at least one shot. However, several other survey organizations have also consistently reported a percentage receiving at least one vaccine shot that since the summer has been lower than the CDC’s numbers.
An important piece of the puzzle are questions that have been raised about the accuracy of the CDC estimates. For example, as explained by The Washington Post, the CDC has reported numbers of seniors receiving at least one vaccine in many states well in excess of the population of seniors in those states. The CDC website now explains that privacy protections limit their ability to link records for first, second, and third doses, limitations that explain why “CDC’s data may over-estimate first doses and underestimate booster doses.”
And a new Bloomberg News report cites state officials who have also concluded that the CDC is counting “too many shots as first doses” that should instead be counted as second doses or booster shots. The results of our surveys – which now allow for an estimate of those who have received booster shots – are consistent with this pattern. While they show a lower estimate than the CDC of the percentage with at least one dose (a divergence which widened significantly since boosters became available in August), and by similar margins, they show a higher estimate of boosters than the CDC reports.
Meanwhile, based on the new three-question set, YouGov’s estimate of the proportion of fully vaccinated Americans over the past six weeks (averaging 67%) is slightly higher than with the original single question approach, and closer to the CDC estimate over the same period (averaging 71%). Our new questions, by getting a count for each respondent of the number of vaccine shots administered, also allows for an estimate of shots per adult, which can be compared to counts published by CDC. Over the past six weeks, the Economist/YouGov survey estimate of the number of shots per adult is only slightly lower (1.57 on average) than the CDC per adult count (1.64 over the same period).
While we found that question wording made little difference in how many people said they'd been vaccinated, we will continue to watch these issues closely and consider if we need further changes to our surveys in order to keep up with changes to the pandemic. When survey researchers present their findings to each other, the line "more research is needed" has become something of a cliche they use to close. In this case, more research is inevitable.
– Kathy Frankovic, Taylor Orth, Linley Sanders, Joe Williams, Ian Davis, Carl Bialik, Delia Bailey, and Doug Rivers contributed to this article.
Methodology: The surveys for the Economist and Yahoo News were conducted by YouGov using nationally representative samples of U.S. adult residents interviewed online between September and December 2021.
Survey experiments were conducted on surveys of U.S. citizens fielded for the Economist between November 6-9 (1,500 respondents; margin of error ±2.7%), November 14-16 (n=1,500; MoE ±2.7%), November 20-23 (n=1,500; MoE ±2.7%) and November 27-30 (n=1,500; MoE ±2.7%) and by YouGov between September 30 and October 8 (n=4,000). Experiments were also conducted on a survey of U.S. adults fielded for Yahoo News between October 1-4 (n=1,640; MoE ±2.7%). These samples were weighted according to gender, age, race, and education based on the American Community Survey, conducted by the U.S. Bureau of the Census, as well as 2016 Presidential vote, registration status, and news interest. Respondents were selected from YouGov’s opt-in panel to be representative of all U.S. adults for Yahoo News, and U.S. citizens for the other surveys.
Estimates of the cumulative percentages of U.S. adults vaccinated are from data published by the Centers for Disease Control and Prevention. Percentages of U.S. adults receiving a COVID-19 booster shot were calculated by dividing the daily cumulative total number of fully vaccinated adults who received a COVID-19 booster (or additional) dose by the total adult U.S. population population implied by the CDC’s daily cumulative percentage of adults, and number of adults, with at least one vaccine dose.
The CDC reports statistics for the United States including vaccinations administered in U.S. territories (Puerto Rico, American Samoa, the Federated States of Micronesia, Guam, the Commonwealth of Northern Mariana Islands, Republic of Palau, the Republic of the Marshall Islands, and U.S. Virgin Islands). YouGov surveys sample from the 50 states and the District of Columbia, but not U.S. territories. As of December 19, CDC statistics indicated a lower percentage of adults with at least one vaccine in the 50 states plus DC (84%) than in the U.S. territories for which data is available (93%), but the proportion of the adult population in the territories is so small (roughly 1% of the total) that their inclusion makes little difference (roughly 0.1%) in the cumulative estimate.
Image: Getty
