Reading results without statistics

An A/B test compares groups of users, and groups differ by chance even when the change does nothing. The statistics on an experiment exist to answer one question: is this difference bigger than chance alone would produce? This page explains how Mitzu answers it, in plain words.

The verdict in one sentence each

Verdict	What it means	What to do
Significant win	The variant really does better on the primary metric. Chance alone would rarely produce a gap this big.	Check the guardrails, then roll it out
Significant loss	The variant really does worse.	Don't ship it. Learn why
No clear winner	Any difference is small enough to be chance.	If the sample plan was reached, the change doesn't matter much; keep the simpler option
Inconclusive: sample too small	No difference was found, but the test was too small to find the lift you were looking for.	Run longer, or accept you won't know
Too few users to compare	Too few users or conversions to compare at all.	Wait, or check the exposure data
Verdict withheld	The test itself is broken: users saw several variants, or the split is off.	Fix the test; don't use these numbers
No verdict yet	The test is still running, or the results don't cover the whole window yet.	Wait for the end; evaluate once more

The badge under the verdict sums it up: Significant win: yes only for a significant win, Significant win: no for a loss, no clear winner or a sample too small. With too few users to compare, or no verdict, there is no badge.

The numbers in a cell

Take a cell that reads:

13.20%
1,320 of 10,000
+10.0%
95% interval +2.2% to +18.3%
p = 0.011
Significant win

• 13.20% / 1,320 of 10,000 — 1,320 of the 10,000 users who saw this variant converted.
• +10.0% — this variant's rate is 10% higher than the control's (if the control converts at 12.00%, this is 13.20%). That's a relative lift, not 10 percentage points.
• 95% interval +2.2% to +18.3% — the true lift is very likely somewhere in this range. It is above zero, so the variant is better; how much better is less certain than +10.0% suggests.
• p = 0.011 — if the variant made no difference at all, a gap at least this big would turn up by chance about 1.1% of the time. Lower is stronger evidence.
• Significant win — the interval stays above zero after accounting for every variant and metric tested.

Plan with the bottom of the interval. If rolling out only pays off with a lift of at least 5%, an interval of +2.2% to +18.3% says the payoff is likely but not assured.

What "significant" doesn't mean

• It doesn't mean big. With enough users, a +0.5% lift can be significant. Ask whether the lift is worth acting on.
• It doesn't mean certain. At 95% confidence, about 1 in 20 tests of a change that does nothing still comes out "significant". That is why a surprising win on a big decision is worth confirming with a follow-up test.
• "Not significant" doesn't mean "no effect". It means this test couldn't tell. Whether that's because there is no effect or because the test was too small is what the sample plan and the Inconclusive: sample too small verdict tell you.

Mistakes that create false wins

Why not stop a test early?

Day by day, the lift of a test swings up and down, especially in the first days when few users are in it. If you look every day and stop the first time the numbers say "significant", you will stop on a lucky swing far more often than 1 time in 20 — in practice, several times more. Mitzu shows no verdict until the test window ends for this reason. Decide the length before the test, and judge it at the end.

Choosing the metric after seeing the results

If a test has ten metrics and you pick whichever one moved, you have run ten tests and kept the best. One of them will often look significant by chance. Choose the primary metric before the test; Mitzu locks it when you schedule.

Slicing until something wins

"It didn't win overall, but it won for mobile users in Germany on weekends." With enough slices, something always wins. Treat such a finding as an idea for the next test, not as a result.

Trusting a fading lift

A change that is new gets attention. If Show details shows the lift was large in the first days and has shrunk since, the long-term effect is likely smaller than the final number. See Metric details over time.

Ignoring the guardrails

A variant can win on its primary metric by doing harm elsewhere: more signups who never activate, more purchases and more refunds. A breached guardrail is a reason to stop and look.

Frequently asked questions

How long should a test run?

At least as long as the sample plan needs, in whole weeks, and at least one full week so every weekday is represented. Set the sample plan and divide the users needed by your daily traffic per variant.

Can I look at the results while the test runs?

Yes: evaluate as often as you like to check that the test is healthy. Just don't act on the numbers until the window ends.

The test is over, so why is there no verdict?

The experiment needs one more evaluation after the end date, so the results cover the last day. Mitzu runs it on its own the morning after the end date and emails the owner when the results are ready. Until then the experiment says Ended · re-evaluate for the final numbers; click Re-evaluate if you don't want to wait.

Why do Mitzu's numbers differ from my experimentation tool?

Small differences are expected. The usual reasons:

• Different metric definitions. Mitzu measures your saved insight, with its filters and conversion window. Your tool measures its own goal events.
• Only after exposure. Mitzu counts what users did after their first exposure in the allocation window.
• Different users. Mitzu counts users with the exposure event in your warehouse, matched on your user identifier. Your tool may count users it assigned, including some who never saw the change, or count devices.
• Users in several variants are counted in each variant they saw in Mitzu.
• Timing. Mitzu's test window follows your workspace's time zone, which may differ from your tool's, and a running test's results include the current day's partial data.

Large differences usually point at the exposure event or the user identifier. Ask your analyst to check the population with Review definition.

What confidence level should I use?

Keep 95% unless you have a reason. Use 99% for decisions that are costly or hard to reverse, such as pricing. Use 90% only for cheap, easily reversed changes where a wrong call costs little.

Should I use a one-sided test?

Only if you would make the same decision whether the variant is worse or merely no better — typically because you only ship it if it wins. A one-sided test needs fewer users, but it can't show that a variant did harm on the primary metric.

A test has two winners. Which one is better?

The verdict names the variant with the larger lift, but the test compared each variant with the control, not with each other. If their intervals overlap a lot, the data doesn't say which is better. Pick on other grounds, or test the two against each other.

What if the result goes against what everyone expected?

Check the health checks and the population first. If the test is healthy, the result is the result. Record it honestly — a surprising result is the most valuable kind, because it is the one nobody would have acted on without the test.