Just a few years ago, it seemed like pollsters possessed a crystal ball. Nate Silver correctly predicted the winner in 49 states in 2008, only to follow it up with a perfect 50 for 50 in 2012. Polling was in its heyday, and it seemed that pollsters had largely cracked the code on how to decipher the political world around them.
That was until 2016. Donald Trump’s unexpected White House victory sent shockwaves throughout the country, but perhaps most devastatingly through the polling world, which had been signaling a surefire win for Hillary Clinton in the months leading up to Election Day. Since then, polling has struggled to regain its footing and accuracy with new issues arising each election cycle. We saw it again in 2020 when polls badly missed on several key Senate races, leading donors to allocate resources to the wrong campaigns. Some of the same issues persisted from 2016 and 2018, but there were also new wrinkles, making it harder for donors to adjust their strategies and make informed decisions.
Top pollsters and polling organizations have been working to address these problems. So where do things stand now, as the 2022 midterms approach, when it comes to the accuracy of polling? That’s an important question for both individual campaign donors and electoral groups as they make decisions about which races to focus their resources on.
Blue Tent spoke with experts in the polling industry and reviewed evaluative reports on the topic to try and get some answers. The spoiler: there are still no concrete answers. Polling is in a transitional state as technology has changed its methodology and the political landscape has changed the variables pollsters need to account for. What’s still true is that polling continues to be an important part of American politics and a necessary factor for donors looking to make smart decisions with their money — but it should never be the only factor. This brief lays out the status of polling today, the efforts being made to improve it, and how donors can use polls wisely.
What Happened to the Polls?
The oversimplified answer for why polling has gotten so difficult in the last ten years is the decline of landline telephones. Landlines were the lifeblood of American polling for decades, providing a uniform medium through which you could conduct a randomized survey through the phonebook and publicly listed contact information. But the Internet and cell phones have dealt the home phone a steady death: In 2004 less than 10 percent of American adults lived in “wireless only” households, meaning they didn’t have a landline. Today, 68 percent of adults have cut the cord.
For pollsters, this has meant declining response rates. As the lead researchers at Pew wrote in 2019, telephone response rates that year were at 7 to 6 percent, compared to 36 percent in 1997. Pew writes that response rates declined with landline rates thanks in large part to spam calls, which have boomed on cell phones in recent years. Lower response rates on phones don’t necessarily mean phone polls are less accurate, but it does make them more expensive because pollsters are trying to reach the same sample size at a much slower pace–which means paying for more people taking more time to make more calls.
All this means pollsters are in a new era after decades of well-established methods and practices. Again, this doesn’t mean online polling methods (which will be detailed later in the brief) are inherently less accurate, but the transitional period does partly account for the rocky road polling has had in the last couple of election cycles.
Trump’s Shock to the System
The infamous whiplash of President Donald Trump’s 2016 victory is of course the landmark event in America shaking its love affair with number whisperers. The 2016 report from the American Association for Public Opinion Research came to some early theories for why polls were so off: the “shy Trump voter” theory — the idea being that supporting Trump was so socially toxic that his supporters were afraid to admit they would vote for him in surveys that many voters changed their mind for Trump late in the race after most polls had already been conducted; and, the most salient, was the education level of survey respondents. AAPOR found most polls did a poor job accounting for adults without a college degree, who tended to support Trump much more than adults with higher education. This kind of demographic miscalculation led many pollsters to start weighing for partisan leanings when conducting polls. Polling issues persisted for the next two election cycles, with pollsters fixing some problems but finding fresh ones.
The 2020 cycle displayed how overly-optimistic polls and misallocated campaign funds go hand-in-hand. More than a dozen Senate races had pre-election polling that undercounted support for the Republican by at least seven percent. That meant in races like Democrat Sara Gideon’s in Maine, donors poured in millions as polls showed her securely ahead of incumbent Republican Susan Collins for months leading up to Election Day. Despite bringing in more than double the fundraising of Collins, Gideon lost by nearly nine points — and nearly a year later still had $10 million of campaign funds unspent. Another famous example was South Carolina Democrat Jaimie Harrison, who Democrats hoped could unseat Republican Senator Lindsey Graham. Polling in the months leading up to Election Day showed Harrison tied or not far off from Graham, inspiring donors to pour in more than $100 million to Harrison’s campaign only to see him lose by 10 points to Graham.
These donations were not solely the result of donors independently looking at polls, of course: Democratic fundraising e-mails and late-in-the-race pushes from Democratic leadership steered donors’ money toward these ultimately doomed candidates. But even the most diligent donor could be forgiven for joining the party’s failures since the available polling seemed to reinforce the party’s marching orders.
What Do Polls Look Like Today?
With polling so up-in-the-air, the onus is now on donors themselves to parse through the polling they see as they make decisions about their money. It can help to start digging more thoroughly into how polling has changed, who’s making the effort to improve, and getting generally more familiar with how polling works. The sources who spoke with Blue Tent for this brief were honest about the industry’s lack of answers, and encouraged by the spirit of experimentation and determination to improve.
Pew writes that there are now two main forms of polls: probability-based and opt-in. Probability-based surveys use phone calls, or else text messages or mailed invitations to take an online survey. These produce better data, but are more time-consuming and expensive, Pew writes. Opt-in surveys, on the other hand, are much cheaper, and are advertised across the Internet. The trouble with them is they create bias since the only people answering the poll are those interested in sharing their opinion, and they are also unable to factor in geographic location — plus they are vulnerable to bots or other interference. Pew also notes that opt-in surveys are often conducted by private companies that don’t reveal their methodology, once again indicating the challenges of the fresh polling landscape.
Cost and geographic targeting issues are amplified at the state level. Smaller races have always been and continue to be much harder to conduct accurate polling on because the sample size is smaller, meaning you have to work harder to reach the right people, and much fewer polling firms have the resources to get adequate data for state polls compared to national ones. House races specifically tend to see turnout that largely aligns with national partisanship numbers for that region since most House candidates don’t have much individual name recognition, but that leaves primary polling for House races in a deeply difficult position.
How to Use Polling as a Donor
It’s very difficult to say definitively that you can trust polling or that you should ignore polling. Ultimately, at least for the next few years as the industry experiments with ways to solve its deeper issues, it will require donors and electoral groups to make case-by-case decisions. Blue Tent hesitates to recommend certain types of polls or even specific pollsters since so much of the industry is in flux and each new election cycle sees different alliances of firms and newsrooms, plus new types of strategies.
An obvious recommendation that Blue Tent heard from all its sources for this brief is to lean on aggregations of polls rather than one survey. The larger aggregation sites, while imperfect, are far more likely to offer you a fair representation of how a race is going than one single poll. Donors should also look at sites that use both polls and other indicators to gauge how competitive different races are, such as 538, Cook Political Report, and Sabato’s Crystal Ball.
Beyond this basic advice, here are some additional suggestions to help donors use polls effectively as they make decisions about their time, money, and resources:
1. Read Carefully, Don’t Just Look at the Numbers
While you shouldn’t expect to become a polling expert, it will help to dig deeper into the background of the polls you’re looking at. Important details to review are who conducted the poll, the methodology of the survey, and the context of other numbers you’ve seen. One important metric to check is the sample size, plus the demographic breakdown of the sample — particularly the partisan breakdown. You should also look at methodology and favor polls using multi-modal survey methods, meaning they’re reaching people through the phone, Internet, and elsewhere. More generally, look to see how transparent the pollsters are with this kind of information. You don’t need to know all the intricacies of “margin of error” or have a statistics background, but see if the poll comes with an explanation of the survey and how they got their numbers. It’s not a guarantee of accuracy, but more candid pollsters at least offer you a look under the hood and can indicate some amount of confidence in their work.
Be sure to also know what question the poll is asking: is this a traditional poll or a forecast? Is this a single poll or an aggregation? If it’s an opinion poll, quite literally what questions were asked of the survey respondents — and was “I don’t know” a response option? Additionally, be curious about the source of the poll: was this conducted by a newsroom in partnership with a university, or is this a private poll from a partisan source that was released publicly? If you’re looking at state-level polling, has this pollster conducted polls in this state before? How accurate were they last time? Taking the extra time to dig into these questions will help you understand what you’re looking at and hopefully inform better decision-making.
2. Keep an Eye on the Big Players
National party organizations make multi-million-dollar decisions each election cycle based on polling information — often from polls the public will never see. Keeping an eye on how places like the DCCC are allocating their money can be a way to peek behind the curtain at what private polling is saying. Of course, this should still be weighted against your individual resources and marginal ability to impact a race — if a candidate is getting flooded with cash, you may want to jump onto supporting a winner but your donation may have a relatively small impact.
Similarly, it can help to try and think like a national political strategist. Try using district-level data from the 2020 presidential race as your benchmark. Weigh the numbers you’re seeing against Biden’s margin of victory or loss in that particular district. (See our recent blog post: Which Democratic Candidates Have a Chance to Win in 2022—and Which Don't?)
This can be helpful in particular when looking at House of Representative race polling since in most House races candidates are unlikely to achieve name recognition that outweighs party affiliation. Finally, always ask yourself, particularly in the case of privately-conducted polls released to the media: why did this get published now? Who conducted this poll and what are they hoping to achieve by letting the whole country see it all of a sudden?
3. Don’t Let Polling Drive Everything
With polling so in flux right now, you shouldn’t be tempted to make massive decisions based on polling. Let’s say it’s August of 2022 and the candidate you’ve been supporting all summer is shown to be down significantly in their race according to the most recent polls: don’t panic! Maybe your candidate may, in fact, be dead in the water, but there’s so much to consider before reaching that conclusion. Blue Tent readers should use their political instincts to assess polling in the context of the race they’ve watched for months. Remember too that polls are just one way to evaluate the strength of a campaign — so are fundraising numbers. There may be situations where contributions from die-hard backers like you could inspire a wave of new funding, which in turn drives media coverage of your candidate which could improve their numbers in the next poll. All of this is to say that poll numbers are not magic, and they are certainly not deterministic.
4. Give More for Long-term Organizing
A good way to avoid the pitfalls of trying to assess polling on particular races is to instead focus your donations on progressive power-building organizations. State-level grassroots groups like Mijente, New Georgia Project, or Pennsylvania Stands Up are a great destination for funds that will go towards building organizing and building capacity that will remain even after Election Day. There are also national groups that support these kinds of organizing efforts. You can learn more about them in Blue Tent’s midterm guidance brief “Giving to Elect Democrats and Build Progressive Power in 2022.”