A definitional guide
Synthetic A/B testing
TL;DR
- Synthetic A/B testing runs proposed site variants against modeled buyer cohorts before any live traffic sees them.
- It decides which variant is most likely to lift conversion in under an hour, with zero customers exposed to losing variants.
- Squoosh matched the live winner in 81% of completed A/B tests in our blind benchmark across e-commerce, SaaS, and fintech.
What is synthetic a/b testing?
Synthetic A/B testing is the practice of evaluating proposed website variants against modeled buyer cohorts — synthetic shoppers — before any real customer is exposed to them. Where a traditional A/B test waits for live visitors to accumulate enough statistical power to declare a winner, a synthetic test runs every variant through the same modeled buyers and ranks them by predicted conversion lift in under an hour.
The synthetic shoppers aren’t random simulations. Squoosh calibrates them from aggregated behavioral signals on your site — funnel position, device mix, intent, category affinity, returning-vs-new — so the cohort running through your variants matches the cohort you’d expect from real traffic, without ever exposing a real customer to a losing variant.
How does it work?
Model the cohort
Squoosh ingests your analytics and order history (Shopify, GA4, Segment, Fullstory) and builds a population of synthetic shoppers that mirror the behavior of your real funnel. No cookies, no PII, no waiting on real traffic.
Run every variant in parallel
Each shopper traverses every variant in a paired test — same brain, different page. The delta in their behavior is attributed to the variant, not to the population mix.
Rank by predicted lift
Squoosh computes conversion rate, revenue per visit, and drop-off points for each variant, then applies McNemar significance to the paired observations. The output is a ranked list of variants with confidence intervals and a recommended winner.
Decide before live traffic sees it
Push the winner straight to your launch workflow (Vercel, Cloudflare, LaunchDarkly, GitHub, webhook), or hand it to your A/B testing tool (Optimizely, VWO, Adobe Target, AB Tasty) for live confirmation. Either way, the losing variants never burn live conversions.
When should you use it?
| Synthetic A/B testing | Traditional A/B testing | |
|---|---|---|
| Time to result | Under an hour | Days to weeks (until statistical power) |
| Live customers exposed to losers | Zero | Half (or more) of test traffic |
| Traffic required | None | Thousands of conversions per variant |
| Best for | Pre-launch screening, low-traffic surfaces, ranking many variants fast | Final confirmation on a single chosen variant |
| Statistical method | Paired-design McNemar test on synthetic cohorts | Frequentist comparison of independent samples |
Frequently asked questions
How is synthetic A/B testing different from a traditional A/B test?
Traditional A/B tests expose real customers to every variant and wait for statistical power. Synthetic tests run modeled buyers — calibrated from your behavioral data — through every variant and rank them in under an hour. No live traffic exposed, no waiting on conversions to accumulate.
How accurate is synthetic A/B testing in practice?
Squoosh matched the live winner in 81% of completed A/B tests in our blind benchmark across e-commerce, SaaS, and fintech flows. We picked a winner before seeing the live result, then compared. The full evaluation report is available on request.
Does synthetic testing replace running a live A/B test?
It replaces the screening step — the part where you exposed losing variants to real customers just to find out they were losers. Many teams use synthetic to rank ten ideas down to one, then run a live confirmation test on just the winner. That cuts live-traffic exposure by 90%+.
Do you collect any personal data from my customers?
No. The synthetic shoppers are modeled from aggregated behavioral signals — device, funnel position, intent — not from individual customer profiles. No cookies, no PII, no tracking real humans. Direct identifiers are never required to run a synthetic test.
What kinds of changes can I test?
Anything that produces a renderable variant of a page: copy, layout, hero, pricing display, urgency banners, social proof, checkout flow, navigation, image vs video. Squoosh runs the same shoppers through each one.
How is significance measured?
Each shopper sees every variant, so the comparison is paired. Squoosh applies McNemar’s test to the paired observations and reports a confidence interval on the predicted lift. Variants whose intervals overlap with control are flagged as inconclusive.
See it on your site
Bring a site change. We'll test it live in 30 minutes — no live traffic exposed.