When pre-election polls missed the mark in 2020, some proclaimed the death of surveys. The focus on accuracy near elections is reasonable, as it provides a unique opportunity to compare a survey estimate to the election result. The ongoing campaign to vaccinate Americans against COVID–19 presents another and very different benchmark available to evaluate survey results, although the comparison highlights an additional important aspect of survey accuracy: how well do the questions measure the behavior or opinion of interest?
Recently, YouGov has been asking Americans about whether they have been or plan to be vaccinated. The results to some questions have been remarkably close to the official statistics released by the Centers for Disease Control and Prevention (CDC). But where the results diverge, they tell a story about how survey researchers can adapt and test questions to get a more accurate result.
In surveys conducted for Yahoo News, YouGov asks a simple, yes-or-no question: Have you been vaccinated for COVID-19? On the last four Yahoo surveys, fielded in March and April, the “yes” response has been very close to, but slightly higher than, the official daily CDC tally of the percentage of U.S. adults who have received at least one dose of a COVID-19 vaccine. In each case, the survey results have been two to three percentage points higher than the official tally, though just within the range of each survey’s reported margin of sampling error.
For example, on the most recent survey, fielded April 27–28, 57% of adults said they have been vaccinated, compared to the CDC report of 54.5% of adults having at least one vaccine shot on April 28. The same survey has a margin of sampling error of +/- 2.8%.
The Yahoo News surveys also include a follow-up question that asks each vaccinated respondent whether they have received one or two shots. The percentage reporting two shots on the Yahoo surveys nearly matches the CDC trend line of the percentage of U.S. adults who are “fully vaccinated.” On the most recent survey, for example, 37% of poll respondents reported having two shots, compared to the CDC report of 37.8% fully vaccinated. On the previous survey, fielded April 6–8, 25% of respondents said they had received two COVID–19 vaccine shots, matching the CDC report of 24.9% of adults fully vaccinated as of April 7.The Economist/YouGov polls use the same data source, sample selection procedure and weighting methodology, but ask about vaccination differently. Starting in late December, the Economist/YouGov survey adapted a question that had been tracked throughout 2020 – “When a coronavirus vaccine becomes available to you, will you get vaccinated?” – by adding the category “I have already been vaccinated” to the previously used answer categories, “yes”, “no” and “unsure.” While results to the question capture the steady upward trend in vaccinations, the percentage who select “already vaccinated” on the Economist question has been significantly lower than the CDC statistics, with the gap widening in recent weeks.
The difference likely occurs because some respondents, who already received a vaccine shot, may not read far enough to see the “already vaccinated” response before clicking “yes.” But there is a slight complication in the population measured: Yahoo News surveys sample all U.S. adults while the Economist surveys sample U.S. citizens.
To be certain, YouGov conducted a series of experiments to compare the two questions, using only samples of U.S. citizens. On the April 17–20 Economist/YouGov survey, a random half of the respondents were asked the yes/no “have you been vaccinated for COVID–19” question, and 53% said yes, just over two percentage points higher than CDC’s report of 50.7% receiving at least one vaccine dose as of April 19.
On the other half-sample, just 40% selected the “already vaccinated” response on the “will you get vaccinated” question, falling well short of the CDC report of 50.7% with at least one vaccine dose.
A week later, YouGov repeated the split-sample experiment on a separate national YouGov survey of U.S. citizens conducted April 24-26, with nearly identical results. One random half sample was asked the “have you been vaccinated” question, and 55% said yes, compared to 53.6% of adults who had received at least one dose as of April 25, according to the CDC. On the other half sample, just 39% chose the “already vaccinated” response to the “when will you get vaccinated” question.
Both experiments confirmed that the questions’ wording and format affected accuracy. The more straightforward “have you been vaccinated” question comes significantly and consistently closer to the official benchmark. However, while the difference is slight and typically falls within each survey’s margin of sample error, the same question also has slightly but consistently overstated the size of the vaccinated population.
There are other complications, however. At first, asking those vaccinated if they have received one shot or two allowed for an accurate estimate of how many are fully vaccinated, since virtually all had received vaccines manufactured by Pfizer or Moderna, which require two shots. However, more Americans are now receiving the Johnson & Johnson vaccine, which requires only one shot. As of May 3, 8% of fully vaccinated Americans had received the single Johnson & Johson vaccine shot.
So last week, as part of a third experiment, a random half sample on the April 25–27 Economist/YouGov survey received an altogether new question:
How would you describe your personal situation regarding COVID–19 vaccines?
- I have received all the injections required to be fully vaccinated against COVID–19
- I have started the vaccination process, but need another shot
- I plan to get vaccinated
- I will not get vaccinated
- I’m not sure about getting vaccinated
On this question, a combined 54% reported they were fully vaccinated (39%) or had started the process but still needed another shot (15%), matching the 53.9% of adults who had received at least one dose as of April 26. The percentage who said they were fully vaccinated (39%) was two percentage points higher than CDC’s report for April 26 (37.0%).
Based on all of these results, we have decided to switch to the new question on whether Americans have been or plan to be vaccinated, featuring five answer categories, on the Economist/YouGov tracking poll.
These results show that modern polling continues to yield good, actionable information but, as always, remains subject to many challenges – in sampling and question wording – that require judgement and experimentation to get right.
See the toplines and crosstabs from this YouGov data analysis
Related: How personal experience shaped public opinion on COVID-19
Methodology: The surveys for the Economist and Yahoo News were conducted by YouGov using nationally representative samples of US adult residents interviewed online between February and April 2021. Survey experiments were conducted on surveys of U.S. citizens conducted for the Economist between April 17-20 (n=1,500; MoE ±2.7%) and April 25-27 (n=1,500; MoE ±2.7%) and by YouGov on April 24-26 (n=1000; MoE ±3.3%). These samples were weighted according to gender, age, race, and education based on the American Community Survey, conducted by the US Bureau of the Census, as well as 2016 Presidential vote, registration status, and news interest. Respondents were selected from YouGov’s opt-in panel to be representative of all US residents.
Estimates of the cumulative percentage of U.S. adults vaccinated published daily by the Centers for Disease Control were compiled by YouGov. These totals for the United States include vaccinations administered in U.S. territories (Puerto Rico. American Samoa, the Federated States of Micronesia, Guam, the Commonwealth of Northern Mariana Islands, Republic of Palau, the Republic of the Marshall Islands, and US Virgin Islands). YouGov surveys sample from the 50 states and the District of Columbia, but not U.S. territories. As of April 15, CDC statistics indicated a higher percentage of adults with at least one vaccine in the 50 states plus DC (50%) than in the U.S. territories (37%), but the proportion of the adult population in the territories is so small (roughly 1% of the total) that their inclusion makes little difference (less than 0.2%) in the cumulative estimate.
Image: Getty
