Polling: The Most Challenging Form of Market Research

Watch any news broadcast this week and you’re likely to hear conversations about the upcoming midterm elections. Despite being over eight months away, the battles for Congress and statehouses across the nation are already well underway – and a major part of the public conversation. During this election season, polling will be a key function of gauging how a particular candidate or topic is doing with the voting public.  Many who utilize polling data live and die by it.  For those who conduct it, it can be one of the most visible, challenging and public aspects of market research because the general public knows how accurate the study/poll was immediately after the fact.  When a poll predicted Hillary Clinton to beat Trump by 5 points on the Monday before the election, we knew by Wednesday how precise it was.

This is rather rare in market research as we typically test an idea and the data is proprietary to the brand/client.  After testing, the idea goes through many changes over time and once it’s actually a product, many other variables like advertising, product placement, distribution and discounts are factors in the success.  Political polling is rare in that you can quickly validate how accurate they are.

Recent high profile misses by the polling industry include:

• 2014 US midterm elections did not predict the Republican landslide that led to majorities in both houses
• 2014 British polls predicted a close election only to see the Conservative party easily win
• 2015 Israeli presidential election that severely underestimated Netanyahu’s popularity/win
• 2016 US presidential polls had Hillary Clinton winning the presidency, only to see Donald Trump become the 46^th president of the United States

These very public misses by the polling industry are a threat to its reputation and validity.  However, these weren’t the first misses by the polling industry.

• In 1936, polls indicated that Republican Alf Landon would win a landslide victory over Democrat Franklin Roosevelt. With 2.4 million respondents this poll had almost zero margin of error from an analytical standpoint. Nonetheless, the poll was wrong as its 2.4 million respondents were not a truly random sample of the US population.
• In 1948, national polls predicted that Republican Thomas Dewey would handily defeat Democrat Harry Truman. In one of the most famous photos of the 20th century, Truman is seen holding aloft a newspaper bearing the headline “Dewey Defeats Truman.” The problem: bad sampling, again.

In 2018, we’re in a similar position to where we were in 1936.  In 1936 the problem with telephone polling was that Republicans were much more likely to have a telephone than Democrats so it over-represented them in the poll.

Political polls never have been easy.  Various items are extremely challenging for political polls:

• Mobile respondents. The rampant increase in mobile device usage (over 350 million in the US) has numerous related and complex issues:
  • Recent regulations make it so that random digit dialers cannot be used to dial a mobile device. An interviewer has to manually dial the number.
  • Interviewers can only speak to the person answering the mobile device
  • What percentage of mobile devices to traditional landlines? Polling companies have to estimate how to distribute surveys by device.  This isn’t easy, especially with the exponential growth.   Also, those that use mobile devices are more likely to be a democrat.
  • More than 60% of adults under the age of 45 only use mobile phones versus just 13% of those ages 65 and older. Hispanics are much more likely to rely solely on their mobile devices than any other race.
• Decline in those that are willing to answer surveys – non response bias. When I was a telephone interviewer in the mid-90s I remember we had a response rate of nearly 50%.  According to the chart below, they are now below 10%.  The fundamental question is “are those who refuse to answer surveys different than those that do?”  We’ve always made an assumption that they are not different but as response rates diminish that assumption is being questioned.
• How to determine a likely voter?
  • Studies suggest that about 75% of people will claim that they will vote in an election. We know from polling data that only about 40-45% of people actually vote.  So, how do you determine which of the 75% that claim that they will vote will actually vote?
  • Companies have tried numerous ways to do this, primarily by asking other questions that strongly correlate to future voting behavior: Past voting behavior, how strongly they feel about certain candidates, etc.  This obviously varies by age which further complicates this.
  • And the analysis of this data is even tougher: You will likely include anyone that says they’re “Extremely Likely to Vote” in your poll, but what about “Very Likely” or those that are “Somewhat Likely” to vote?
• Questionnaire wording:  Statisticians and pollsters have argued for many years what the proper wording for a political poll should be.   Do you ask simply how likely they are to vote and who they plan on voting for?   Do you ask if they voted in previous elections?  How passionate are you pro/against certain candidates?   Many models (some outdated) are being used today.
• Are certain candidates under-represented in polls because of social-desirability bias?

For our next presidential election it may be that the Republicans will have more phones than Democrats or are more likely to answer the phone than Democrats.  Or polling companies will not dial enough mobile devices as the science behind polling isn’t advancing as fast as mobile adoption.

Regardless, lots of smart statisticians and analysts have based their entire career on polling will be trying to figure this out over the next year.  Some will be better than others – some will just get lucky.  But, remember this is not an exact science and there are a lot of factors to consider when being critical of the polling industry.

Want to learn how EMI can help with your political polling?  Download our brochure.

References:

http://www.nytimes.com/2015/06/21/opinion/sunday/whats-the-matter-with-polling.html

http://www.huffingtonpost.com/brian-stryker/can-we-trust-polls-withou_b_4880127.html

http://www.huffingtonpost.com/2012/05/15/poll-response-rates_n_1518501.html