Squoosh AI
  • Blog
  • Partners
Sign inBook a demo→
Squoosh AI
  • Blog→
  • Partners→
  • Sign in→
Book a demo→
← BlogResearchNovember 23, 2026

Squoosh Bench: UX-1 on 318 real A/B tests

Squoosh Bench tests whether an AI system can call how a real A/B test ended. On 318 completed tests, Squoosh UX-1, the model behind the village, called 267 outcomes correctly (83.96%). The strongest of three general-purpose AI models called 196 (61.64%). Of the 126 variants UX-1 favored, 118 won their live test.

RP
Rohith Pasula
Research
5 min read
Squoosh Bench: UX-1 on 318 real A/B tests

We believe the A/B test is one of the best quantitative ways there is to measure something qualitative: how people respond to what you put in front of them. People rarely say what they think of a website. They show it, by staying or leaving, and an A/B test counts it.

Squoosh is built on that belief. The Squoosh village sends AI people, the villagers, through both versions of a page and forecasts how the live test would end, so a team can get a forecast for every idea it has, not only a live result for the few it has the traffic to test. Squoosh UX-1 is the model behind the village.

Squoosh Bench is how we check the village against the real thing. It scores an AI system against the recorded outcomes of A/B tests that have already finished. Today we are publishing how UX-1 scored on 318 of them, alongside three general-purpose AI models.

318
Completed A/B tests, outcomes withheld
267 of 318
Outcomes called by Squoosh UX-1
83.96%
Outcome accuracy, against 61.64% for the next system
93.7%
Of the variants UX-1 favored won their live test

How UX-1 scored

UX-1 called 267 of the 318 outcomes: 83.96%. Qwen3 Next 80B called 196 (61.64%), GPT-OSS 120B called 187 (58.81%) and Gemini 3.5 Flash called 162 (50.94%). UX-1 got 71 more tests right than the best of the three general-purpose models.

Outcome accuracy on Squoosh Bench

Share of 318 completed A/B tests each system called correctly

  1. UX-1267 of 318
    Outcome accuracy 83.96%

    71 more tests than the next system 22.33 points ahead

  2. Qwen3 Next 80B196 of 318
    Outcome accuracy 61.64%
  3. GPT-OSS 120B187 of 318
    Outcome accuracy 58.81%
  4. Gemini 3.5 Flash162 of 318
    Outcome accuracy 50.94%
0%43.08% Picking the variant every time100%
Outcome accuracy. The share of the 318 tests where a system’s call matched the recorded result: an original win, a variant win or an inconclusive test. A missing or invalid call counts as wrong. The vertical line marks a rule that picks the variant every time, which is right on 137 of the 318. From Table 4 of the paper.

A live test can end three ways, and in these 318 a variant win was the most common. A rule that picks the variant every time would be right on 137 of the 318 tests, 43.08%. UX-1 was right on 130 more.

The benchmark

Every task in Squoosh Bench is an A/B test that has already finished: two versions of a real web page, and the result recorded from real visitors. We screened 412 tests from private partners and scored the 318 that ran longer than 21 days. They come from more than 40 sites in seven industries, among them retail, telecom, travel, financial services and software.

In those 318 tests the variant won 137 times, the original won 113 times and 68 ended inconclusive. A system has to say which of the three happened. It sees both pages, the goal of the test and the plan for running it. It is not told which page is the original, and the recorded result stays hidden until its call is locked.

Squoosh UX-1 is the model that drives the villagers. We built it by fine-tuning an open-source base model on data that records how people use websites, so that it captures the choices they make: whether they scroll through a listing, open a product page or leave before converting. For every test, UX-1 sent 1,000 villagers through the original page and 1,000 through the variant, then returned one call, an estimated lift and a short explanation.

Three general-purpose models took the same 318 tests: Qwen3 Next 80B, GPT-OSS 120B and Gemini 3.5 Flash. Every system worked from the same frozen copy of each page, with the same device settings and the same goal.

Across all three outcomes

UX-1 was even across the three outcomes. It called 94 of the 113 tests the original won, 118 of the 137 the variant won and 55 of the 68 that ended inconclusive. That last group was the hardest for every system. The general-purpose models called between 31 and 36 of the 68.

Squoosh Bench results

318 completed A/B tests · lift error on the 289 tests every system returned a number for

Squoosh Bench results
MeasureUX-1Qwen3 Next 80BGPT-OSS 120BGemini 3.5 Flash
Outcome accuracy83.96%61.64%58.81%50.94%
Balanced accuracy83.40%60.25%57.11%50.07%
Original wins called83.19%62.83%59.29%51.33%
Variant wins called86.13%64.96%63.50%53.28%
Inconclusive tests called80.88%52.94%48.53%45.59%
Same call in either page order94.65%83.02%79.25%53.14%
Lift error (lower is better)1.433.814.123.27
Results by measure. Balanced accuracy is the average of the three rows beneath it. Lift error is the average gap between the forecast lift and the recorded lift, in points of relative lift. From Tables 4, 5 and 7 of the paper.

A forecast should also not change because two pages traded places. Every system ran every test in both orders. UX-1 returned the same call both times on 301 of the 318 tests. Qwen3 Next 80B held on 264, GPT-OSS 120B on 252 and Gemini 3.5 Flash on 169.

Which tests to run first

The practical question for a team is which ideas deserve live traffic. UX-1 favored the variant in 126 of the 318 tests, 39.6% of them. In the live tests, 118 of those variants won: 93.7%. Four lost to the original and four ended inconclusive.

How the live tests ended, by UX-1’s call

One square per completed A/B test · 318 in all · colored by the recorded live result

UX-1 favored the variant
126 of 318 tests
118 variant won4 original won4 no clear winner
UX-1 did not favor the variant
192 of 318 tests
19 variant won109 original won64 no clear winner
UX-1’s picks. Each square is one of the 318 tests, colored by its recorded live result and grouped by whether UX-1 favored the variant. From Table 6 of the paper.

A team that ran UX-1’s picks first would have tested 126 ideas instead of 318 and found 118 of the 137 winning variants, 86.1% of them. The other 19 winners were among the tests UX-1 did not pick, so the forecast is a way to decide what to test first, not what to throw away.

What we believe

Human feedback is qualitative. It is how a page feels to the person looking at it: whether it is clear, whether it is worth their time. The A/B test is one of the best quantitative ways anyone has found to capture that, because it does not ask people for an opinion. It counts what they do.

That is what we are trying to do here. If AI people are going to stand in for real ones, the test they have to pass is the one real people already took. On 318 of those tests, UX-1 gave the same answer 267 times. That is why we are so excited about this research.

What comes next

We are working with our partners on a plan to open-source the Squoosh Bench A/B testing framework. We will share more in January, including how other companies can submit their systems here.

Read the paper

The paper has the full method, every table and the scoring rules.

Squoosh Research · November 2026
Squoosh Bench: Evaluating UX-1 on 318 Historical Website A/B Tests

A retrospective evaluation of website A/B test outcome prediction: Squoosh UX-1 and three general-purpose AI systems on 318 completed experiments.

Rohith Pasula, Johari Dramiga, Viraj Shah, Suzan Halim and Tyler Dooskin. Squoosh AI. arXiv:2611.25200.
Read the paper (on arXiv, opens in a new tab)→

Keep exploring

  • What is synthetic A/B testing?
  • See the Squoosh AI benchmark
  • Villagers are not your customers, and they stay out of your analytics
  • Why bundling changes into one test backfires

The Squoosh Dispatch

New research,
straight to your inbox.

Deep dives, release notes, and benchmarks. No promos, no tracking beyond the clicks.

Squoosh AISquoosh AI

We simulate how humans interact with digital experiences, so you can see what works before it goes live.

Product

  • Overview
  • How it works
  • Benchmark
  • Your data

Company

  • Partners
  • Careers
  • Press

Resources

  • Blog
  • Documentation
  • Support

Legal

  • Terms
  • Privacy
  • Trust center
© 2026 Squoosh, Inc.