Synthetic data promises to address market research’s most persistent obstacles: data scarcity, privacy compliance, and cost efficiency. Organizations exploring this technology need to understand both its potential and the key challenges in implementing synthetic data before incorporating synthetic respondents into their research strategy. Data quality concerns prompted EMI Research Solutions to conduct systematic testing of multiple synthetic data providers, comparing their outputs against actual human respondents across identical questionnaires and quotas. The findings revealed significant data inconsistencies across providers.

Explore EMI’s approach to data quality, which includes industry-leading survey fraud detection techniques to protect against bots, duplicates, and fraudulent responses.

What Synthetic Data Claims to Deliver

Synthetic data refers to artificially generated information designed to replicate the statistical properties of real-world datasets. In market research, this translates to AI-generated “respondents” who answer surveys, rate products, and express preferences.
The value proposition includes unlimited respondents for hard-to-reach demographics, reduced privacy concerns, lower research costs, improved feasibility for difficult quotas, and faster project timelines. These benefits address genuine constraints that researchers face around sample availability and budget limitations. However, the performance of synthetic data in practical applications reveals substantial gaps between these claims and actual outputs, exposing fundamental challenges in synthetic data accuracy and reliability.

Methods of Using Synthetic Data in Market Research

Organizations employ synthetic data in different ways depending on their research objectives:
  • Boosting involves adding synthetic respondents to existing real respondent data to increase overall sample size. This approach typically supplements a smaller real sample with synthetic data to achieve target quotas without relying entirely on artificial respondents.
  • Augmenting uses synthetic data to fill specific gaps in real datasets, such as underrepresented demographics or behavioral segments. Rather than replacing real data, augmentation supplements it with synthetic observations to improve representativeness across predefined characteristics.
  • Digital Twins generates fully synthetic respondent profiles that mimic real consumer populations based on learned patterns from training data. This method creates complete artificial respondents rather than supplementing existing real data. Digital twins attempt to replicate the full behavioral, demographic, and attitudinal profiles of actual consumers through generative AI models.

How Synthetic Data Gets Generated

Generating synthetic data relies on generative AI architectures (large language models, generative adversarial networks, or diffusion models) trained on existing data to produce new “observations” that mirror real patterns.
EMI’s 2024 testing provided multiple vendors with two waves of research-on-research data to program their AI engines and create synthetic profiles mimicking U.S. consumers. Each received identical questionnaires and demographic quotas. The test design was straightforward: if synthetic data accurately replicates real respondent patterns, outputs should closely match a third wave of actual human responses collected simultaneously.

Three variables determine synthetic data quality, and each introduces potential variation:

  • The AI model architecture influences what patterns can be learned and how well rare cases are preserved. Different vendors use different systems with varying capabilities.
  • The training data provided determines what synthetic outputs can represent. If that data contains bias or gaps, synthetic outputs will reflect these limitations.
  • The prompts and parameters used influence how the model generates responses. Even identical models produce different outputs based on instructions and constraints.

One important limitation emerged immediately: Vendor 3 could only select demographic breakdowns, not the attitudes and behaviors associated with those demographics. This fundamentally limits what their synthetic data could represent.

Where Synthetic Data Breaks Down: EMI's Findings

EMI’s 2024 testing evaluated digital twin type synthetic data. The findings reflect challenges specific to this generative approach to creating synthetic respondents.

Homeownership: Measuring Provider Consistency

EMI tested synthetic providers on homeownership rates among U.S. adults. In Q4 2024, Federal Reserve data established the benchmark at 65.7%. Actual human respondents in EMI’s study reported 54.2%—an 11.5 percentage point difference that reflects known sampling dynamics in online research.

Synthetic providers produced a 39-point range:

  • Vendor 1: 33% (33-point gap from Federal Reserve; 21-point gap from organic)
  • Vendor 2: 53% (within one point of organic respondents)
  • Vendor 3: 72% (closest to Federal Reserve but 18 points from organic)
  • Vendor 4: 55% (within one point of organic respondents)

Two vendors approximated real respondent patterns. Two others produced results that differed substantially from the Federal Reserve benchmark and organic respondents.

Brand Ratings: Distribution Patterns and Bottom-Box Representation

Brand perception measurement revealed notable differences in how synthetic providers handled rating scales. Among actual human respondents rating Coca-Cola on a four-point scale, 45% rated it Excellent, 33% Very Good, 15% Fair, and 6% Poor.

Synthetic providers showed different patterns:

  • Vendor 3 generated 59% “Very Good” ratings—26 points higher than organic respondents. Additionally, 0% of their synthetic respondents rated Coca-Cola as “Fair” or “Poor.”
  • Vendor 1 produced 47% “Very Good” (14 points above organic) and 0% “Poor” ratings.
  • Vendor 2 came closest to organic responses, with a maximum five-point difference across rating options.
  • Vendor 4 could not answer brand rating questions at all.
Most synthetic providers produced bell curve distributions even when real human responses followed different patterns, suggesting that generative models optimize for certain statistical distributions rather than behavioral accuracy. The absence of bottom-box scores in multiple vendors presents a measurement issue, as understanding negative sentiment provides information about brand positioning and competitive vulnerabilities.

Purchase Intent: Variation in Concept Testing Results

EMI tested a fictional wireless charging product using a 10-point scale. Real consumers showed 53% top-4-box purchase intent.

  • Vendor 3 produced 0% top-4-box intent, showing no interest in the product despite using the same concept description and demographic profile as other providers.
  • Vendor 1 followed a bell curve distribution, peaking at 7-8 and declining at 10, a pattern that differs from typical consumer evaluation patterns.
  • Vendor 2 matched organic respondents closely, with only a four-point maximum difference.
  • Vendor 4 could not provide purchase intent data for this product but could for a different concept, indicating inconsistent processing capabilities.

For a theme park concept, Vendor 4’s responses showed a 36-point difference at one data point, with top-2-box scores 21 points lower than organic. The synthetic data over-represented middle-scale responses while under-representing extremes.

Behavioral Questions: Middle-Scale Over-Representation

EMI asked respondents to report daily screen time on a four-point scale. Vendor 4’s synthetic data showed a 9-point average difference from organic respondents, over-reporting the “7+ hours” category by 14 percentage points.

Synthetic providers consistently over-represented middle categories while under-representing extremes. This reflects how generative models weight frequently occurring responses more heavily than outliers.

In market research, extreme cases often provide important information. Heavy category users, strong brand opponents, and price-insensitive consumers represent segments that influence category dynamics. When synthetic data systematically reduces representation of these groups, it alters the market picture being measured.

Discover how the best survey fraud detection software protects research quality in traditional panel sampling.

Why Synthetic Data Produces Systematically Flawed Outputs

Distribution Drift and Over-Smoothing

Generative models learn the probability density function of training data, estimating the likelihood of different response combinations, and generate new observations that follow those patterns. This process tends to produce outputs that cluster around average values while reducing the frequency of extreme cases.

Real consumer populations include substantial variation across behaviors and attitudes. This variation reflects actual market heterogeneity. Synthetic data generation processes tend to reduce this variation in favor of more uniform distributions.

Bias Amplification Rather Than Correction

Organizations sometimes consider synthetic data as a solution for addressing sample representation gaps. However, if an existing sample includes 100 respondents from a particular demographic showing certain patterns, generating 1,000 synthetic respondents based on those patterns does not improve representativeness. It replicates the existing response distribution while creating the appearance of increased statistical precision.

Small samples produce estimates with wider confidence intervals that reflect measurement uncertainty. Synthetic data generation masks this uncertainty with artificially narrow confidence intervals rather than addressing its source.

Correlation Preservation Challenges

Real human behavior involves multidimensional relationships: income correlates with brand consideration but varies by education level, geographic region, and life stage. Generative models attempt to preserve these relationships, but accuracy varies depending on correlation strength and complexity in the training data.

The result can be synthetic data that matches individual variable distributions while producing combinations of characteristics that occur rarely or not at all in actual populations. Several vendors could not provide respondent-level data with demographic breakdowns, preventing verification of whether synthetic respondents represented plausible profiles or statistical constructs.

Incomplete and Inconsistent Response Capability

Multiple vendors could not process certain question types. Vendor 4 provided data for some questions but not others. Vendor 1 could not provide gender-level breakdowns. Vendor 3 could not incorporate behavioral variables into profile generation. These limitations affect research applications where consistent measurement across all variables matters for analysis.

The Three Components That Determine Quality Synthetic Data

EMI’s research indicates that synthetic data quality depends on three interdependent factors:

The AI model or engine

Model architecture determines what patterns can be learned, how correlations are preserved, and how rare cases are handled.

The training data

The quality and representativeness of training data directly affects what synthetic outputs can represent. Model sophistication cannot compensate for biased or incomplete training data.

The prompts and parameters

Instructions, constraints, and prioritization of statistical properties all influence generation outputs.

In EMI’s testing, even when vendors received identical training data and quotas, outputs varied substantially because model architectures and generation approaches differed. This variability means that organizations cannot reliably predict synthetic data quality without extensive validation against real-world benchmarks.

Evaluating Synthetic Data: What Organizations Should Know

Organizations exploring synthetic data should approach it with rigorous validation protocols. At minimum, this includes:
  • Validation against real-world benchmarks – Synthetic outputs must be continuously tested against actual respondent data to identify distribution drift, correlation breakdowns, and systematic biases.
  • Transparency in generation methods – Document the AI model used, training data sources, parameter settings, and any constraints applied during generation. This documentation enables assessment of where biases may originate.
  • Multi-vendor comparison – As EMI’s research demonstrates, synthetic data quality varies dramatically between providers. Testing multiple vendors against the same inputs reveals which systems produce outputs closer to real respondent patterns.
  • Respondent-level verification – Confirm that synthetic profiles represent plausible combinations of demographic and behavioral characteristics rather than statistical artifacts.
  • Question-type consistency checks – Verify that providers can process all questionnaire formats your research requires, as incomplete response capability limits analytical options.
However, even with rigorous safeguards, EMI’s testing shows that synthetic data continues to differ systematically from organic respondents in response distributions, correlation preservation, and representation of extreme values. The variability between providers means organizations cannot reliably predict synthetic data quality without extensive validation, reducing many of the efficiency gains that make synthetic data appealing.

Why Real Respondents Provide Reliable Measurement

For over 25 years, EMI has studied how sample sources differ and how those differences affect research quality. Our research-on-research program examines panel bias, data quality, and representativeness across hundreds of studies and millions of responses. This same methodology applied to synthetic data reveals consistent patterns: synthetic respondents show systematically different response distributions compared to real human participants.
Real people provide authentic variation in opinions, preferences, and behaviors. They express contradictory views, give responses that don’t follow expected patterns, and show the full range of human decision-making. This complexity reflects actual market conditions. Synthetic data generation smooths this complexity into more uniform patterns that align with statistical expectations rather than behavioral reality.

EMI's Approach: Strategic Sample Blending for Reliable Insights

Sample sources vary in their characteristics and biases. This is a reality EMI has documented extensively through our research-on-research program. Our strategic sample blending methodology addresses this variation by intentionally combining three or more high-quality sources in controlled proportions, with no single source exceeding 50% allocation.

Our patented IntelliBlend® approach takes this further by blending sample sources, including traditional panels and select non-traditional sources, in an intentional and controlled manner to deliver the most representative and accurate demographic, behavioral, and attitudinal data. This reduces the influence of individual source characteristics while maintaining authentic response variation from real human participants.

We apply consistent quality standards to every data source in our network. Our Partner Assessment Process evaluates recruitment methods, validation procedures, and data quality measures. Only 30% of evaluated panels meet our standards and enter our network of 150+ global partners.

Our proprietary SWIFT platform provides real-time quality monitoring, fraud detection, and digital fingerprinting. Combined with our Quality Optimization Rating system, which evaluates sample quality across pre-study, in-study, and post-study phases, we maintain data integrity throughout the research process.

This quality framework, built on 25+ years of industry expertise and 12+ years of research-on-research, delivers reliable insights based on authentic human responses.

Ready to discuss how strategic sample blending can support your research objectives? Request a consultation with our sample experts.
Request a Consultation

Frequently Asked Questions

Synthetic data refers to artificially generated responses created by AI models designed to simulate human survey participants. Vendors use algorithms to produce “respondents” based on patterns learned from existing data, rather than recruiting actual people.

EMI’s testing found that synthetic providers produce inconsistent, distorted results across homeownership rates, brand perceptions, and purchase intent. Synthetic data lacks authentic human nuance and behavioral unpredictability that real market research requires.

Synthetic data tends to over-represent average responses while under-representing extreme opinions, shows inconsistent ability to preserve demographic and behavioral correlations, replicates patterns from training data including any biases, and produces distributions that differ systematically from real respondent patterns.