When measuring public opinion on current events, pollsters sometimes just want to know what side you’re on. We occasionally write questions that try to capture opposing sides of an ongoing political or public debate, asking which statement “is closer to your opinion,” even if the competing arguments are neither perfectly parallel nor mutually exclusive. A new survey experiment conducted by YouGov demonstrates how such questions can sometimes pose a difficult choice for respondents and, as a result, produce highly variable findings: because two options both can be appealing, at least in part, a small shift in opinion or difference in question wording can move a large group from one side to the other.
We saw this phenomenon play out in our recent attempt to gauge where Americans stand in the debate over whether to ease COVID-19 restrictions. Like other pollsters, we wanted to know if people were ready to try to return to their pre-pandemic lives or if they thought societal interventions remained necessary.
In December, a question asked by the Republican polling firm Echelon Insights posed a choice between continuing to treat COVID-19 “as an emergency” with “vaccine requirements, masking and other restrictions” implemented “until cases fall to zero or a negligible amount” versus treating COVID-19 as “an endemic disease that will never fully go away” requiring only “vaccines and other treatments to reduce the severity of illness.” Registered voters nationwide preferred the “endemic disease” argument over the “emergency” argument, 54% to 36%. (See full text at the end of this article.)
The result, which was published and shared widely on social media, helped spur a discussion among pundits and public health experts about whether COVID politics had “shifted, even decisively,” especially “among the folks to take COVID seriously” or “among Democrats rather than between Democrats.” To try to get a better handle on that speculation, YouGov asked simpler versions of similar questions on surveys conducted for Yahoo News in January, February, and March.
One question in particular asked respondents to choose one of two statements as “closer” to their opinion on COVID-19: In late January, 46% of U.S. adults selected “we need to learn to live with it and get back to normal,” and 43% selected the alternative, “we need to do more to vaccinate, wear masks and test.”
In two subsequent surveys, the proportion who prefered to live with it and return to normal has risen slightly, from 46% to 51% in late February, while the number who preferred to do more to mask, test and vaccinate declined from 43% to 36%.
Over the same time span, YouGov and other pollsters took alternative approaches, producing even more widely varying results:
- In late January, the Monmouth University Polling Institute found 70% of U.S. adults agreed with the statement, "It’s time we accept that Covid is here to stay and we just need to get on with our lives” (28% disagreed).
- In early February, the Axios/IPSOS poll offered four alternative statements: A total of 50% of U.S. adults preferred an easing of restrictions, either “open up and get back to life as usual with no mandates or requirements” (21%) or “move towards opening up, with precautions” (29%). A total of 44% preferred one of the remaining choices, either to “mostly keep precautions and requirements in place” (23%) or “increase mask mandates and vaccine requirements” (21%).
- In mid February, the Economist/YouGov poll found 77% of U.S. adult citizens were “ready to return to normal from the COVID-19 pandemic.”
- In late February, the Washington Post/ABC News poll found 58% of U.S. adults considered “trying to control the spread of the coronavirus, even if it means having some restrictions on normal activities” more important than “having no restrictions on normal activities, even if it hurts efforts to control the spread of the coronavirus” (38%).
There are obvious differences in wording and question structure that may explain some of the considerable variation in these results. Also, looking at just our own question, some respondents might want to agree with elements of both statements, while others might agree with some parts of one statement but not others.
Given the importance of the ongoing public debate about easing COVID-19 restrictions, we conducted an experiment on theYouGov’s U.S. News Survey to better understand the complexity of opinion underlying this particular question. We asked U.S. adult citizens if they agree or disagree with each specific element of our two-statement question regarding COVID-19.
Consider first the two elements of the statement, “we need to learn to live with it and get back to normal.” Better than four out of five (83%) agree that “we need to get back to normal,” and almost as many (78%) agree that “we need to learn to live with it.” However, the similarity of the aggregate numbers conceals bigger differences among individual respondents. While the vast majority (82%) of those who agree we need to “get back to normal” also agree that we need to “learn to live with it,” those who disagree about getting back to normal are more divided – just 56% agree we need to learn to live with it and 44% disagree.
In other words, “getting back to normal” does not always mean “learning to live with it” and vice versa (for the statistically minded, the correlation in responses to the two questions is 0.24).
Now consider the three elements of the other statement in our original question, “we need to do more to vaccinate, wear masks and test.” Again, very similar percentages of our respondents agree about each separate element: “we need to do more to wear masks” (55%), “we need to do more to vaccinate” (57%), and “we need to do more to test” (59%).
Here too, there are differences among individual respondents, but the correlations are stronger. For example, 82% of those who agree with the need to do more to vaccinate also agree we need to wear masks, and 79% of those who disagree about vaccinations also disagree about mask wearing (the correlation between these two items is 0.61 and the correlations between each combination of the three items ranges between 0.61 and 0.63).
But the similarity in opinion is not lockstep. While 41% of all adults agree with the need to “do more” on all three restriction questions (on vaccination, masks, and testing), 16% agree with only two of the three and another 14% agree with just one. So there are many respondents who might agree with part of the combined statement, “we need to do more to vaccinate, wear masks and test,” but not all of it.
Since our original question used the phrase “do more” regarding these restrictions, we also asked whether “we need to keep existing precautions in place,” and found essentially the same level of agreement (58%) as for “doing more” on masks (55%) and vaccinations (57%) and testing (59%).
The bigger issue is that many agree with the sentiments expressed in both sides of our original two-category question, which we subsequently repeated in this survey (51% of U.S. adult citizens selected “we need to “learn to live with it and get back to normal,” while 39% selected “we need to do more to vaccinate, wear masks and test”).
Not surprisingly, between 90% and 95% of those who choose the “need to do more to vaccinate, wear masks and test” response also agree with each element of that statement. Similarly, between 91% and 97% of those who choose “we need to learn to live with it and get back to normal” also agree with each of the elements of that statement.
However, among those who choose the statement “we need to do more to vaccinate, wear masks and test”, two thirds (66%) also agree “we need to get back to normal“ and 64% also agree ”we need to learn to live with it."
Meanwhile, about a third of those who choose the statement “we need to learn to live with it and get back to normal” also agree that we need to keep existing precautions in place (31%) and do more to test (36%), vaccinate (31%), and wear masks (27%).
And that’s the nub of the challenge in asking and interpreting this kind of question. Many respondents want to agree with parts of both statements. Consider the overlap of just two even simpler statements regarding COVID-19: “We need to learn to live with it” and “we need to keep existing precautions in place.” In our test, 40% of U.S. adults agree with both statements, while 36% want to learn to live with it but disagree about keeping precautions in place, and 16% want to keep precautions in place and not learn to live with it.
This is not to say that a survey should never ask respondents to choose among competing ideas that may be less than mutually exclusive. We routinely ask respondents to choose which is “most important” among competing priorities, and choosing one of two potentially appealing statements as “closer” to one’s own opinion is similar. More to the point, political debates, and political campaigns, involve competing arguments that often bundle together multiple ideas. There is value in testing which argument has more appeal.
But it is also important to understand when people see merit in both sides of a highly charged argument. One clue is when small differences in wording on this type of question produce highly variable results, but it is also critical to ask additional questions to uncover the aspects of an argument that have broad appeal, even to those on the other side.
These kinds of questions become even more valuable when we keep asking them. However appealing both options are, and however hard a choice it is to select only one, movement toward one of them over time suggests that in the competition of ideas, one side is gaining ground.
See the toplines and crosstabs for this U.S. News Poll
YouGov Methodology: This U.S. News survey was conducted by YouGov using a nationally representative sample of 1,000 U.S. adult citizens interviewed online between February 8 and February 12, 2022. This sample was weighted according to gender, age, race, education based on the 2018 American Community Survey, conducted by the U.S. Census Bureau, as well as news interest and 2020 Presidential vote (or non-vote). Respondents were selected from YouGov’s opt-in panel to be representative of all U.S. citizens. The margin of error is approximately 3% for the entire sample.
Yahoo! News/YouGov Methodology: The results from Yahoo! News surveys conducted by YouGov cited in this article are based on nationally representative samples of U.S. adults interviewed online between January 20-24, February 4-7, February 24-27 and March 10-14, 2022. These samples were weighted according to gender, age, race, and education based on the 2018 American Community Survey, conducted by the U.S. Census Bureau, as well as 2020 Presidential votes (or non votes), and voter registration status. Respondents were selected from YouGov’s opt-in panel to be representative of all U.S. adults.
Echelon Insights question: The full text of the Echelon Insights question quoted in this article is as follows. “Which of the following best describes how you think we should approach COVID-19 moving forward? COVID-19 should be treated as an endemic disease that will never fully go away, with seasonal waves like the flu, that we should respond to with vaccines or other treatments to reduce the severity of illness OR COVID-19 should continue to be treated as an emergency, and in response to rising case numbers we should implement vaccine requirements, masking, and other restrictions until cases fall to zero or a negligible amount.”
