My Big 4 Takeaways from SampleCon 2022

SampleCon 2022 was just a couple of weeks ago, but it has felt like I’ve needed the last 2 weeks to digest the information. Pasadena was a terrific location, and it was fun to get out of Cincinnati and visit as well as catch up with many of our partners. It has felt like years since I have seen many of them in person!

Now that I am back in Cincinnati and have had time to organize all my notes, here are a few of my key takeaways from SampleCon 2022.

We the People – State of the Industry

War on Talent

We are all hiring, and some company strategies are perceived as going as far as using acquisitions as a form of hiring strong talent.

Built with a “people centric” mentality, businesses grow when they build the best teams. There is an old school mentality that companies feel the employees owe them something. However, the pendulum has swung, and the ball is now in the hands of the talent. Companies need to sell themselves as a great place to work over asking potential hires to sell themselves.

We need premium pay for premium talent and performance. We need to build better training programs for hiring and training for a full-time job. Investments in continued training past the onboarding stage, committing to regular reviews, and offering goal-based pay increases are key. It is also crucial to elevate existing staff compensation as we onboard new talent at current market competitive wages to ensure an even playing field to avoid damaging morale of tenured team members.

Culture and people are key to growing business. Asking if potential hires “fit” our culture is the wrong mentality. Attracting new talent is about adapting the company culture, not maintaining old culture. Culture is easy when revenue is up, but how are we managing culture in a volatile ecosystem?

Battle for Preserving our Assets

“It’s the people that matter. Without the respondent, we have nothing.” 

End clients and insights teams are frustrated that they are spending half of their time on identifying bad behaviors and removing bot data. We are focusing less and less on user experience and focusing more on adding additional quality controls and trap questions in surveys. In doing so, we are increasing lengths of interviews and consequently tossing out the baby with the bath water. With inconsistency in quality reviewing best practices, the loyal respondents providing valid data are suffering with poor survey designs and face a higher risk of losing the already substandard compensation for their time.

Unfortunately, if the supply is there to fill poorly crafted experiences and the “someone else will do it if we don’t” mentality perpetuates, the demanders of research won’t change the research designs to be respondent first. If we don’t fix the issues with UX, quality, and incentives willingly and proactively, there will soon be external regulations that will force it upon our industry.

Data Quality is Still a Headlining Topic

This year’s SampleCon presented different perspectives on the data quality conundrum. It is refreshing to not just talk about fraud and bots and how quality needs to improve – we all know this, we agree on this, and of course there was a lot of content around new tools and methods for technology-based protections and safeguards. I was motivated by another people-first approach and the opportunity to take a closer look at user experience and the impact it has on data. The panel of speakers from across the industry shared how poor user experiences are leading to poor quality data. Bad experience = bad database (panel) = bad results.

For instance, the long path to get to a survey where a respondent may get routed to 4-7 different surveys before they finally land on one where they qualify inevitably impacts their mood along the way. We know respondent attitudes can impact results, something that my colleague Brian Lamar is validating with his ongoing Research-On-Research.

The long-term impacts of the poor user experience not only alter the feedback the respondent gives, but also discourages participation in further research opportunities. In this regard, panel managers are not solely responsible for the rise in bots and fraud despite how they have been vilified. Bots don’t care about UX, but human respondents do. If we continue to accept these poor experiences and push them out with little to no incentive, we are making it impossible to retain engaged participation.

Takeaway from observations:

Remember back when aggregation was a dirty word? I do because we received shot after shot from competitors for being consultants that promoted the necessity of unbiased blending to ensure reliable results. Now, there is a ton of incest among panels. Those same competitors that were taking shots at aggregation are now selling more of their competitor’s sample than they are of their own “proprietary” supply.

As part of my role ensuring quality while vetting and onboarding new suppliers, I am always surprised to see how many panels share exclusively other panel companies as references when I ask who currently buys from them. I’m mostly shocked because the references that they are giving rarely share openly that their feasibility is now primarily outsourced when our request is for proprietary sample only.

Most panel companies are now heavy buyers, likely due to the surge in demand over the past few years coinciding with dwindling participation rates, panel membership interest declining, and supply recruitment costs increasing. I see panels building teams of buyers as deep as their team of sellers. Where you think your sample is coming from may not be where you’re buying it from (or where they are buying it for that matter.) Therefore, now more than ever, as a buyer of sample, we must DEMAND TRANSPARENCY. Disregard for adequate vetting in the aggregation process, lack of strategy behind blending, and biased sample planning leads to riskier data, continued respondent abuse, and more easily hacked surveys by fraudsters. The solution in this case is to work with trusted, proven partners who you can work with as extensions of your team that prioritize the health and well-being of our entire eco-system of the insights industry.

If you were not able to attend SampleCon, and would like a custom recap, click the button below.