The Poll Party E2 – David Dutwin of AAPOR and SSRS
August 26, 2019
Intellicast S2E29 – Thoughts on the Second Half of the Year
September 3, 2019
One of the documents most utilized by researchers to select a sample provider is the ESOMAR 28. However, be careful when evaluating possible providers using this document. Most of you are likely very aware of the “ESOMAR 28 Questions to Help Buyers of Online Samples” document, but to those that aren’t, here is some background.
According to its website, “ESOMAR is a not-for-profit organization that promotes the value of market, opinion and social research and data analytics. We’ve been providing ethical and professional guidance and advocating on behalf of our global membership community for over 70 years.”
The 28 Questions to Help Buyers of Online Samples Guide is designed to provide a standard set of questions a buyer of sample can utilize to understand the credibility of sample providers. Many industry thought leaders came together to create this document and it is widely used today. I don’t think a sample provider exists that hasn’t created this document, at least in traditional online quantitative marketing research.
This document includes important questions surrounding recruiting and incentive methods, quality practices, and panel management, which are all integral when determining which sample provider to partner with. However, we have noticed that some panel companies have been utilizing their ESOMAR 28 as another piece of sales collateral, rather than providing direct answers to the questions.
An example of this occurs quite often in Q2, which reads “Please describe and explain the type(s) of online sample sources from which you get respondents. Are these databases? Actively managed research panels? Direct marketing lists? Social networks? Web intercepts (also known as river) samples?”
We understand that sample providers want to keep proprietary recruiting methods from becoming public, but answering the question like the response above can be problematic.
Another example is Q3, which reads, “If you provide samples from more than one source: How are the different sample sources blended together to ensure validity? How can this be replicated over time to provide reliability? How do you deal with the possibility of duplication across sources?” The variation in data coming from different sources has been well documented. The overlap between panel providers can be significant.
There are numerous other examples of this; and unfortunately, the ESOMAR 28 document has become a selling piece being created by the Marketing department rather than an information document created by a sampling methodologist. Therefore, be skeptical of each answer provided to you.
A few additional tips when evaluating sample providers:
- Ask lots of questions, especially clarification questions
- Expect and demand transparency
- Request proprietary sample only
- Build a relationship with these providers as they can change – mergers, acquisitions, recruiting changes, etc. You should have ongoing discussions to obtain any changes that may affect the panel, and subsequently your survey data.
For more information on the best tips on how to find a sample partner, click here.
