Sample plan
A test that ends too early can miss a real improvement and report No clear winner. The
sample plan says how many users each variant needs to detect the smallest lift you care about,
and tracks the test against that number while it runs.
The plan is optional, but without it Mitzu can't tell a real "no difference" from a test that was too small to see one.
Setting a plan
Smallest lift worth detecting. The smallest relative change on the primary metric you would act
on. 5% means a 3.00% order rate rising to 3.15%. Pick the change that would justify the cost of
rolling the variant out, not the change you hope to see.
Baseline rate. The primary metric's rate without the change, for conversion metrics only. Leave it empty to use the control's observed rate once results come in. Per-user and ratio metrics always use the control's observed value and spread.
Fill in the lift alone, or both. A baseline rate without a lift is refused with
Set the smallest lift worth detecting, or clear the baseline rate.
How the required sample is worked out
Mitzu plans for an 80% chance of detecting the lift if it is real, at the confidence level and sidedness in the statistics settings. When the multiple-comparison correction is on, the plan accounts for every comparison it covers, so a test with three treatments and two secondary metrics needs more users per variant than a simple A/B test.
A few things that move the required sample:
| Change | Effect on the users needed |
|---|---|
| Halve the smallest lift worth detecting | about four times as many |
| A lower baseline rate | more |
| 99% instead of 95% confidence | about 50% more |
| One-sided instead of two-sided | about 20% fewer |
| More treatments or secondary metrics | more, when the correction is on |
Before the test starts, divide the required sample per variant by the number of users who reach the tested experience per day and per variant. That's the shortest test window that can work. If it is months, test a bigger change or a metric closer to it.
Following the plan
Once results come in, the plan is checked against the smallest variant, because the comparison is only as strong as its weakest side.
- The Sample against plan health check reads, for example,
The smallest variant has 3,210 of 5,000 users (64.2%).It passes when the plan is reached. - While the test runs, the experiment also says how long it is likely to take:
Sample 64.2% of plan · 3,210 of 5,000 per variant · at least about 6 more days at current traffic.
When the test window ends with no clear winner, or too few users, and the plan not
reached, the verdict is Inconclusive: sample too small: the test was too small to rule out the lift
you planned for. See Results and verdicts.
When no sample can be planned
| Message | What it means |
|---|---|
Sample plan needs a baseline rate | No baseline rate is set and there are no results yet to observe one |
Sample plan set · waiting for primary metric results | The plan uses the control's observed value, which needs a first evaluation |
No sample can be planned: the baseline rate is 0% or 100%, or the lift takes it past 100% | Lower the lift or check the baseline rate |
No sample can be planned: the control's value is 0 or does not vary between users | The per-user metric has no spread to plan from |
The sample plan is part of the locked definition: set it while the experiment is a draft.