Like most researchers, you’ve spent time investigating the differences in survey data from one wave to the next. You reviewed demographic data or even attitudinal behavior data to see why these differences exist. Keeping your sample as consistent as possible from wave to wave is ideal, but can be challenging in our world with low incidence studies, a need for higher base sizes, hard to find targets, etc.
At EMI, we’ve noticed that another source of differences in data can be attributed to the actual sample source. We conducted numerous research on research studies among 30+ different US sample partners to understand how sample sources differ. Although targeting and balancing were replicated across all sample partners, there were significant differences in the data for all sources. The chart below illustrates some of the differences we saw testing various concepts. You’ll notice that top box concept rating is very different based upon the sample partner you select.
There are a host of reasons sample can look different from source to source or even wave to wave with a single source such as
- Recruitment approach
- Opt-in process and verification
- Incentive structure (points, sweep, cash, etc.)
- Sample source composition
- Panel management (tenure, profiling, targeting, a community or just visitors…etc.)
- Sample delivery mechanism (direct email/router/prescreen)
One way of reducing the bias of one source is to systematically blend multiple sample sources. At EMI we have been blending sample for 15 years and our “IntelliBlend®” blending product provides a much more controlled approach to sampling. We help identify which studies benefit and will put together a comprehensive sample plan utilizing multiple sample sources that ensure consistency. In this way, you can feel more confident about comparing results from one test to another. We are also able to help you transition your existing sample plan to a better, more realistic long-term option.