In hindsight, Patrick Murray says, the vacation question should have been a tipoff. It should have signaled that something was wrong with polling in the presidential race.
For years, pollsters have asked Americans if they have vacations planned, one of many questions intended to take the temperature of the country. But when Murray’s team at the Monmouth University Polling Institute asked in June, the results seemed off: 75% of Democrats said they were planning a summer vacation before the coronavirus struck, compared with just 43% of Republicans.
That was an absurd gap for a question that usually elicited little partisan difference.
Murray surmised why: Respondents knew the next question would be about if they had to cancel those plans due to the pandemic. And Democrats were eager to blame President Donald Trump for mishandling it, while Republicans didn’t want to make the president look bad.
“It was the most bizarre thing I’ve seen in polling," said Murray, Monmouth’s polling director. “It should have been a clue to all of us that there are a number of things out there that we are measuring that are related to Trump that we’re not getting an honest answer on.”
It was a similar story on the economy: No matter the fluctuations in the real world, Democrats were giving more negative reviews and Republicans more positive.
“It didn’t react to changes in the economy and the unemployment rate like it should have," Murray said. "It was just simply tied into opinion of Donald Trump. That should have also indicated to us that maybe we weren’t measuring the president accurately.”
The answers were so unenlightening that he stopped asking the question.
After a second consecutive presidential election in which public opinion surveys significantly understated Trump’s support, Murray and other pollsters are grappling with what went wrong. Is the issue specific to Trump? Can their work accurately gauge the pulse of the country, even beyond elections?
“The horse-race question about elections is the least-valuable thing that polling does as an industry,” Murray said. “It is going to be a much more fundamental question about ‘Can we accurately measure the American public right now given how divided we are as a nation?’”
This year was supposed to be different.
When state-level polls in Pennsylvania and other key battlegrounds misread Trump’s support in 2016, explanations and adjustments followed. The 2016 race had been volatile throughout, with a large number of undecided voters. There was news — the renewed FBI investigation of Hillary Clinton’s emails — that drove a late surge for Trump. And among white voters, educational attainment became a new dividing line.
So pollsters put more focus this time on whether respondents had gone to college, and weighted their samples to account for the difference in vote preferences (college-educated white voters overwhelmingly support Democrats, while white people who didn’t attend college heavily favor Trump).
Chris Borick, who runs the Muhlenberg College poll, said that’s actually not a terrible gap. His final poll had Biden leading by 5, but with a margin of error of plus or minus 5.5 percentage points, a 1 point Biden win is well within his probabilities.
Murray had said before the election that anything from a narrow Trump win to big Biden victory seemed possible in Pennsylvania.
“The problem is it’s two cycles in a row that generally the errors have tended to break in the same direction and against the same candidate,” Borick said.
And while national polls in 2016 were fairly accurate, in 2020 they vastly overstated Biden’s advantage. Some key states that polls projected to be close, like Texas and Ohio, produced comfortable Trump wins, while some states where Biden was seen with huge leads, most notably Wisconsin, were neck-and-neck.
This soon after the election, pollsters are still combing through the evidence, but one early theory involves the “shy” Trump voter.
For years, some Republicans have argued that polls understate support for the president because Trump backers were reluctant to talk to pollsters, or admit their support. There were little hard data to support this idea, though, and many other more concrete explanations for the 2016 polling misses, so the theory was often dismissed.
Now, there’s some consideration that it might be right.
One reason: After pollsters adjusted their methods, the industry largely got the 2018 midterm election right. The difference? Trump wasn’t running that year.
“I don’t want to try to attribute it all to some kind of President Trump effect, but there seems to be some different behavior in elections over the last four years when he’s on the ballot,” Borick said. “I can’t perfectly diagnose that, but I can see some of the patterns there.”
Murray also said that the “shy” Trump voter “seems to be the leading indicator.”
“It also suggests that we’ve probably been understating Donald Trump’s approval by about 2 points for the last four years,” Murray said.
Murray said the “shy” Trump effect may have grown since 2016. The president was more unknown then, but since then his behavior has come to be seen as unacceptable by many.
“One of the things that we know from polling is that we have a much harder time accurately measuring issues where there is a social desirability concern, that people don’t want to be seen in a certain way,” Murray said.
Neil Newhouse, a Republican pollster based in Virginia, saw signs of this in a poll he conducted on Election Day. Of the 360 Trump voters he surveyed, 19% said they had kept their vote secret from most of their friends.
Many said they feared being called a racist, ostracized, or physically attacked, he found.
“Many Americans want to keep their opinions to themselves for fear of being judged by others who may disagree,” Newhouse said.
Among the answers he said he received:
“I got called a white supremacist and a racist, so I kept it to myself so I wouldn’t hear those words.”
“I don’t want to get beat up.”
“I have a lot of liberal friends. They get angry.… `Oh my God I can’t believe you’re voting for him, what’s wrong with you? Oh, he committed fraud, he’s a racist, he’s homophobic.'”
Trump’s attack on the very idea of polling, like his barrages against many institutions, may also have influenced the results, Newhouse said.
Trump’s tactic “actually discourages his supporters from participating,” Newhouse said. “There’s simply a decent chunk who believe that all polls are fake and are used by the media to make their case. They just won’t respond.”
Internal GOP polling in Pennsylvania showed Trump with more support than public polls suggested — though not by much, and even those internal surveys understated the president’s final share of the vote, according to one Pennsylvania Republican who saw the internal surveys.
The source, who was not authorized to publicly discuss the polling, suggested that blue-collar workers who form Trump’s base of support may be less likely to answer the phone and sit for a lengthy questionnaire about politics.
Murray, of Monmouth, said that reaching a representative sample of working-class voters has always been a challenge, however, and that pollsters adjust their results to reflect that difficulty.
Some public polls did project a closer race. Susquehanna Polling & Research, based in Harrisburg, found an extremely tight race in its final survey, with Trump at 49% and Biden 48%. The eventual Biden win was well within its margin of error.
Murray and Borick stressed that polls are not meant to be precise measurements of the race’s outcome. They are intended as broad indicators of the contest, such as whether it’s close and which way it is trending. While most attention focuses on the top-line numbers, the margins of error suggest a range of possible states of the race.
“Every election has polling error,” Murray said. “In most cases we just don’t notice because the polling is basically in close enough to what the outcome is.”