Political Polling: Reliability, Challenges, & Changes

Is Political Polling Reliable?

By now, you’re probably aware of the ongoing debate of the reliability of political polling. From before the 2014 midterms to the 2020 presidential election, polling data has been under fire. This year will likely be another year of contention and questioning the validity of polling questionnaires as 2022 is a midterm election year in the US.

Many who utilize polling data live and die by it. For those who conduct this type of research, it can be one of the most visible, challenging, and public aspects of market research because the public knows how accurate the study/poll was immediately after the fact. When a pre-election poll predicted Joe Biden’s lead over Donald Trump in the 2020 presidential election by 4 points, we knew by Wednesday how precise it was.

This is rather rare in market research as we typically test an idea, and the data is proprietary to the brand/client. After testing, the idea goes through many changes over time, and once it’s a product, many other variables like advertising, product placement, distribution, and discounts are factors in the success. Political polling is rare in that you can quickly validate how accurate they are.

Recent high profile misses by the polling industry include:

• 2014 US midterm elections did not predict the Republican landslide that led to majorities in both houses
• 2014 British polls predicted a close election only to see the Conservative party easily win
• 2015 Israeli presidential election that severely underestimated Netanyahu’s popularity/win
• 2016 US presidential polls had Hillary Clinton winning the presidency, only to see Donald Trump become the 46^th president of the United States
• 2020 US presidential polls overstated Biden’s voter support, predicting a large win when it ended up becoming a relatively close race in some states

These very public misses by the polling industry are a threat to its reputation and validity.  However, these weren’t the first misses by the polling industry.

In 1936, polls indicated that Republican Alf Landon would win a landslide victory over Democrat Franklin Roosevelt. With 2.4 million respondents, this poll had almost zero margin of error from an analytical standpoint. Nonetheless, the poll was wrong as its 2.4 million respondents were not a truly random sample of the US population.

In 1948, national polls predicted that Republican Thomas Dewey would handily defeat Democrat Harry Truman. In one of the most famous photos of the 20th century, Truman is seen holding aloft a newspaper bearing the headline “Dewey Defeats Truman.” The problem: bad sampling, again.

In 2022, we’re in a similar position to where we were in 1936. In 1936, the problem with telephone polling was that Republicans were much more likely to have a telephone than Democrats, so it over-represented them in the poll. Today, we know that left-leaning folks tend to take more surveys, meaning we often have a non-response bias from right-leaning Americans.

Political polls never have been easy.  Various items are extremely challenging for political polls:

• 1) Mobile respondents – the rampant increase in mobile device usage (85% of Americans own a smartphone now, up 35% from 2011) has numerous related and complex issues:
  • Recent regulations make it so that random digit dialers cannot be used to dial a mobile device. An interviewer has to manually dial the number.
  • Interviewers can only speak to the person answering the mobile device.
  • The percentage of mobile devices compared to traditional landlines is disproportionate. Polling companies must estimate how to distribute surveys by device. This isn’t easy, especially with the exponential growth. Also, those that use mobile devices are more likely to be Democrats.
  • 100% of adults under the age of 50 use mobile phones versus just 92% of those age 65 and older. Hispanics are more likely to rely solely on their mobile devices than any other race.
• 2) The decline in those that are willing to answer surveys – causing non-response bias. When I was a telephone interviewer in the mid-90s, I remember we had a response rate of nearly 50%. They are now below 10%. The fundamental question is “are those who refuse to answer surveys different than those who do?” We’ve always made an assumption that they are not different but as response rates diminish that assumption is being questioned.
  • How to determine a likely voter?
    • The US census recorded that roughly 67% of people were registered to vote in the 2020 election. However, only about 61% of people voted.  So, how do you determine which of the 67% will vote?
    • Companies have tried numerous ways to do this, primarily by asking other questions that strongly correlate to future voting behavior: past voting behavior, how strongly they feel about certain candidates, etc. This obviously varies by age and changes as election day draws closer which further complicates this.
    • Further, the analysis of this data is even tougher – you will likely include anyone that says they’re “Extremely Likely to Vote” in your poll, but what about “Very Likely” or those that are “Somewhat Likely” to vote?
• 3) Questionnaire wording: statisticians and pollsters have argued for many years what the proper wording for a political poll should be. Do you ask simply how likely they are to vote and who they plan on voting for? Do you ask if they voted in previous elections?  How passionately are you pro/against certain candidates? Many models (some outdated) are being used today.
• 4) Certain candidates are likely under-represented in polls because of social-desirability bias.  For example, in the 2020 presidential election, there was an underrepresentation of Trump voters in polling data. Is this because those voting for Trump were less likely to share their voter intent due to social-desirability bias?

For our next presidential election, it may be that Republicans will be more likely to answer political surveys than Democrats or more people age 65+ will be using mobile devices.

Regardless, a lot of smart statisticians and analysts who have based their entire careers on polling will be trying to figure this out for years to come. Some will be better than others – some will just get lucky. But remember this is not an exact science and there are a lot of factors to consider when being critical of the polling industry.

References:

• http://www.nytimes.com/2015/06/21/opinion/sunday/whats-the-matter-with-polling.html
• http://www.huffingtonpost.com/brian-stryker/can-we-trust-polls-withou_b_4880127.html
• http://www.huffingtonpost.com/2012/05/15/poll-response-rates_n_1518501.html
• https://www.pewresearch.org/internet/fact-sheet/mobile/
• https://www.pewresearch.org/methods/2021/03/02/what-2020s-election-poll-errors-tell-us-about-the-accuracy-of-issue-polling/
• https://www.census.gov/data/tables/time-series/demo/voting-and-registration/p20-585.html