ESOMAR 28 Updates: What They Mean and Why You Should Care- Part 3- Sampling and Project Management

ESOMAR introduced an updated version of their Questions to Help Online Sample Buyers (formerly known as the ESOMAR 28).We have been exploring the changes to this document and what it means for researchers. In Part 3 of our series, we will be continuing our deep dive into the changes; this time looking at the Sampling and Project Management section.

Sampling and Project Management

This section gives buyers details on the processes, procedures, and methods a sample provider utilizes to provide respondents to your survey. It covers everything from invitation methods, respondent profiling, feasibility, back-ups, and much more. This will help the buyer understand the potential benefits and/or biases that may exist utilizing a specific panel.

What’s Changed

Briefly describe your overall process from invitation to survey completion. What steps do you take to achieve a sample that “looks like” the target population? What demographic quota controls, if any, do you recommend?

The first part of this question is a rewording of a question from the ESOMAR 28– the real change is the addition of the second half of the question regarding the “looks like” target population. With it being more common for projects’ target audiences to match census or a specific target audience, the need for tools to help manage this has grown. Sample providers can use this question to let sample buyers know the methods and tools they currently employ to ensure specific make-up of target audiences within projects.

What profiling information do you hold on at least 80% of your panel members plus any intercepts known to you through prior contact? How does this differ by the sources you offer? How often is each of those data points updated? Can you supply these data points as appends to the data set? Do you collect this profiling information directly or is it supplied by a third party?

This is a completely new addition to the ESOMAR questionnaire. Over the last couple years, many organizations have been looking to add additional data to the insights they collect through their surveys. Whether it is profiled information or adding other third-party info, the ability to get a more well-rounded view of respondents and the data they provide has become critical to many organizations. Buyers can use the responses to this question to better understand the profiling information a specific panel can provide on its panelists and if there is other information they can provide through third-party appends.

What information do you need about a project in order to provide an estimate of feasibility? What, if anything, do you do to give upper or lower boundaries around these estimates?

The first part of this question is the same one that appears in the ESOMAR 28 – it is the second part of the question that is new.  With the wild fluctuations in feasibility that the industry experienced in 2020, this question is good for buyers to understand the feasibility of a provider. It can allow them to plan for additional partners and/or back-ups in case a primary partner falls short.  This prep work at the beginning of a project can prevent a lot of headaches and scrambling when a project is trying to get closed and additional providers are needed to be brought on to top up sample.

What do you do if the project proves impossible for you to complete in field? Do you inform the sample buyer as to who you would use to complete the project? In such circumstances, how do you maintain and certify third-party sources/sub-contractors?

This question goes hand-in-hand with the previous question, but with the focus more on transparency. As many of us know, sample aggregating and sample blending are now commonplace with sample providers. Whether they are transparent about it or not is a much different question.  Buyers should use this question to understand how specific suppliers deal with bringing in additional providers to supplement their panels or how providers blend panels together.  They should be looking for answers that talk about the transparency of using additional suppliers at the bidding stage of the project rather than in-field or post-fielding of a study. 

What information about a project is given to potential participants before they choose whether to take the survey or not? How does this differ by the sources you offer?

This is an interesting addition to the new ESOMAR questionnaire.  In the past, the sample industry has never really talked about the information a potential survey respondent is given about a potential study. Buyers may not think this question is important, but the information that a potential respondent gets about a survey (or the potential reward) may influence if they respond or not, which leads to a potential bias. 

Do you allow participants to choose a survey from a selection of available surveys? If so, what are they told about each survey that helps them to make that choice?

This question presents the same potential bias issue that the previous question explores as well. Buyers should look at this question and the previous question together to better understand if there might be a potential respondent bias created due to the amount of information on the study given to a potential respondent.

Do you measure participant satisfaction at the individual project level? If so, can you provide normative data for similar projects (by length, by type, by subject, by target group)?

This is another new question in the questionnaire. Participant satisfaction is a good metric for panels to use to help understand the willingness of a respondent to take future surveys.  This is important to panels because it takes resources to recruit panelists and if one no longer wants to take surveys and is replaced, their demographic make-up is most likely different. 

Buyers can look at this to see if they are able to get the respondent feedback to better improve their processes and survey design to help ensure that respondents provide high-quality insights.

Do you provide a debrief report about a project after it has completed? If yes, can you provide an example?

This is another new question that has been added to the new questionnaire. Buyers should expect sample providers to be able to list their standard reports and metrics they make available after a project. If a provider cannot, buyers should be wary.

Check back with us next week for the next blog where we will be looking at the Data Quality and Validation section and the changes that have happened.

Missed any other parts of this series? Check out the list below for all the other entries in this series:

Part 1

Part 2