ESOMAR 28 Updates: What They Mean and Why You Should Care- Part 4- Data Quality and Validation

ESOMAR introduced an updated version of their Questions to Help Online Sample Buyers (formerly known as the ESOMAR 28). Over the last couple of weeks, we have been exploring the changes to this document and what it means for researchers. In Part 4 of our series, we will be continuing our deep dive into the changes; this time looking at the Data Quality and Validation section.

Data Quality and Validation

This is a newly defined section that helps give prominence to the measures a sample provider is taking to validate their panelists and to ensure they provide quality data. While many providers take action to help ensure data quality before, during, and after a study, this section really focuses on the in-study data quality. 

What’s Changed:

How often can the same individual participate in a survey? How does this vary across your sample sources? What is the mean and maximum amount of time a person may have already been taking surveys before they entered this survey? How do you manage this?

Much of this question remains the same from the previous version of the questionnaire – the difference lies in what has been added, namely the number of times a person could have taken the survey and how a provider manages it. The additions really focus on tracking studies since the targeting remains the same and is on a set time interval, so it is good to understand how many times a potential respondent could provide feedback over the course of the tracker. 

How do you manage source consistency and blend at the project level? With regard to trackers, how do you ensure that the nature and composition of sample sources remain the same over time? Do you have reports on blends and sources that can be provided to buyers? Can sources be appended to the participant data records?

This is a totally new addition to the ESOMAR questionnaire. As sample aggregating and sample blending has become not only more commonplace, but almost standard in online quantitative research, clients need to understand not only the sources they engage, but any potential sources those panels bring on as well. This helps with not only transparency of where your data is coming from but can also help sample buyers understand any biases that may arise because of specific partners, as well as an easy way to avoid duplication. 

Another key aspect sample buyers should look for is how a provider maintains the consistency of their sample blends. This is crucial, especially with tracking studies, to eliminate any outside biases to be confident any changes you are seeing in the data are due to market forces and not sample plan changes.  For more information on how EMI manages consistency and blends, please visit us here (link?)

Please describe your participant/member quality tracking, along with any health metrics you maintain on members/participants, and how those metrics are used to invite, track, quarantine, and block people from entering the platform, router, or a survey. What processes do you have in place to compare profiled and known data to in-survey responses?

This is another new addition of the ESOMAR questionnaire. Many different sample providers use different algorithms, flags, or reasoning to either confirm or block respondents in-survey. This question can help sample buyers understand how a specific sample provider does it.  It also gives the opportunity for sample providers to compare the data quality measures in which different sample providers use to find one that matches their needs.

Overall, ESOMAR is doing a great job of trying to get the sample providers to provide key metrics with consistent measures so buyers can easily compare and contrast. 

For work where you program, host, and deliver the survey data, what processes do you have in place to reduce or eliminate undesired in-survey behaviors, such as (a) random responding, (b) Illogical or inconsistent responding, (c) overuse of item nonresponse (e.g., “Don’t Know”) (d) inaccurate or inconsistent responding, (e) incomplete responding, or (f) too rapid survey completion?

This question really gets into the specific processing and/or technology that a sample provider utilizes to monitor and reduce undesired behavior that respondents may exhibit when taking a survey.  The methods a sample provider details should help a sample buyer understand how much data cleaning they may need to undertake if they commission a specific provider for a project.

Check back with us next week for the next blog where we will be looking at the Policies and Compliance section and the changes that have taken place.

Missed any other parts of this series? Check out the list below for all the other entries in this series:

Part 1 – Company Profile

Part 2 – Sample Sources and Recruitment

Part 3 – Sampling and Project Management