We believe the A/B test is one of the best quantitative ways there is to measure something qualitative: how people respond to what you put in front of them. People rarely say what they think of a website. They show it, by staying or leaving, and an A/B test counts it.
Squoosh is built on that belief. The Squoosh village sends AI people, the villagers, through both versions of a page and forecasts how the live test would end, so a team can get a forecast for every idea it has, not only a live result for the few it has the traffic to test. Squoosh UX-1 is the model behind the village.
Squoosh Bench is how we check the village against the real thing. It scores an AI system against the recorded outcomes of A/B tests that have already finished. Today we are publishing how UX-1 scored on 318 of them, alongside three general-purpose AI models.
How UX-1 scored
UX-1 called 267 of the 318 outcomes: 83.96%. Qwen3 Next 80B called 196 (61.64%), GPT-OSS 120B called 187 (58.81%) and Gemini 3.5 Flash called 162 (50.94%). UX-1 got 71 more tests right than the best of the three general-purpose models.
Outcome accuracy on Squoosh Bench
Share of 318 completed A/B tests each system called correctly
- UX-1267 of 318Outcome accuracy 83.96%
71 more tests than the next system 22.33 points ahead
- Qwen3 Next 80B196 of 318Outcome accuracy 61.64%
- GPT-OSS 120B187 of 318Outcome accuracy 58.81%
- Gemini 3.5 Flash162 of 318Outcome accuracy 50.94%
A live test can end three ways, and in these 318 a variant win was the most common. A rule that picks the variant every time would be right on 137 of the 318 tests, 43.08%. UX-1 was right on 130 more.
The benchmark
Every task in Squoosh Bench is an A/B test that has already finished: two versions of a real web page, and the result recorded from real visitors. We screened 412 tests from private partners and scored the 318 that ran longer than 21 days. They come from more than 40 sites in seven industries, among them retail, telecom, travel, financial services and software.
In those 318 tests the variant won 137 times, the original won 113 times and 68 ended inconclusive. A system has to say which of the three happened. It sees both pages, the goal of the test and the plan for running it. It is not told which page is the original, and the recorded result stays hidden until its call is locked.
Squoosh UX-1 is the model that drives the villagers. We built it by fine-tuning an open-source base model on data that records how people use websites, so that it captures the choices they make: whether they scroll through a listing, open a product page or leave before converting. For every test, UX-1 sent 1,000 villagers through the original page and 1,000 through the variant, then returned one call, an estimated lift and a short explanation.
Three general-purpose models took the same 318 tests: Qwen3 Next 80B, GPT-OSS 120B and Gemini 3.5 Flash. Every system worked from the same frozen copy of each page, with the same device settings and the same goal.
Across all three outcomes
UX-1 was even across the three outcomes. It called 94 of the 113 tests the original won, 118 of the 137 the variant won and 55 of the 68 that ended inconclusive. That last group was the hardest for every system. The general-purpose models called between 31 and 36 of the 68.
Squoosh Bench results
318 completed A/B tests · lift error on the 289 tests every system returned a number for
| Measure | UX-1 | Qwen3 Next 80B | GPT-OSS 120B | Gemini 3.5 Flash |
|---|---|---|---|---|
| Outcome accuracy | 83.96% | 61.64% | 58.81% | 50.94% |
| Balanced accuracy | 83.40% | 60.25% | 57.11% | 50.07% |
| Original wins called | 83.19% | 62.83% | 59.29% | 51.33% |
| Variant wins called | 86.13% | 64.96% | 63.50% | 53.28% |
| Inconclusive tests called | 80.88% | 52.94% | 48.53% | 45.59% |
| Same call in either page order | 94.65% | 83.02% | 79.25% | 53.14% |
| Lift error (lower is better) | 1.43 | 3.81 | 4.12 | 3.27 |
A forecast should also not change because two pages traded places. Every system ran every test in both orders. UX-1 returned the same call both times on 301 of the 318 tests. Qwen3 Next 80B held on 264, GPT-OSS 120B on 252 and Gemini 3.5 Flash on 169.
Which tests to run first
The practical question for a team is which ideas deserve live traffic. UX-1 favored the variant in 126 of the 318 tests, 39.6% of them. In the live tests, 118 of those variants won: 93.7%. Four lost to the original and four ended inconclusive.
How the live tests ended, by UX-1’s call
One square per completed A/B test · 318 in all · colored by the recorded live result
A team that ran UX-1’s picks first would have tested 126 ideas instead of 318 and found 118 of the 137 winning variants, 86.1% of them. The other 19 winners were among the tests UX-1 did not pick, so the forecast is a way to decide what to test first, not what to throw away.
What we believe
Human feedback is qualitative. It is how a page feels to the person looking at it: whether it is clear, whether it is worth their time. The A/B test is one of the best quantitative ways anyone has found to capture that, because it does not ask people for an opinion. It counts what they do.
That is what we are trying to do here. If AI people are going to stand in for real ones, the test they have to pass is the one real people already took. On 318 of those tests, UX-1 gave the same answer 267 times. That is why we are so excited about this research.
What comes next
We are working with our partners on a plan to open-source the Squoosh Bench A/B testing framework. We will share more in January, including how other companies can submit their systems here.
Read the paper
The paper has the full method, every table and the scoring rules.
