Core principles
Experiments is built around a handful of rules. Each one closes off a common way an A/B test gives the wrong answer. Knowing them explains most of what the product lets you do, and what it stops you from doing.
1. Mitzu analyses tests, it doesn't run them
Your experimentation tool or your own code assigns users and serves the variants. Mitzu reads what they recorded. Nothing you do on an experiment — saving, scheduling, evaluating, finalizing — changes the test your users see.
What it means for you: stop, extend or ramp a test in the tool that runs it, then update the test window in Mitzu if you need to.
2. Only exposed users count, from the moment they were exposed
A metric is measured for each variant on the users who have the exposure event in the test window, and only on what they did after their first exposure. A purchase made the week before someone saw the new checkout says nothing about the new checkout.
What it means for you: the conversion rates on an experiment are lower and cleaner than the same insight on a dashboard, which counts everyone and everything in its date range. See Metrics.
3. The analysis is decided before the results are seen
While an experiment is a draft you can change anything. Once you schedule it, the population, metrics, statistics settings, sample plan, variants and test window are locked. Changing them means moving the experiment back to draft, which hides the results computed so far.
What it means for you: pick the primary metric, the guardrails and the confidence level when you plan the test, not after you have seen which metric moved. Picking the metric that happened to win is the most common way to declare a false win. See Editing and locking.
4. One primary metric decides the test
Every experiment has exactly one primary metric, and the verdict is about that metric alone. Secondary metrics are supporting evidence. Guardrails are metrics that must not get worse.
What it means for you: if two things matter equally, run the test on the one you would act on, and keep the other as a secondary metric. See Metrics.
5. No verdict until the test window has ended
While the test window is still open, the experiment is Running · incomplete and the Results
card shows the numbers so far with the heading Incomplete: no verdict yet. A verdict appears
only once the window has ended and the results cover all of it.
What it means for you: watching the numbers during a test is fine; acting on them is not. Lifts are noisy in the first days and checking repeatedly until something looks significant makes false wins much more likely. See Metric details over time.
6. A broken test gets no verdict
Health checks run on every result. When users were seen in more than one variant, or the traffic split is clearly not what the test was set up with, the verdict is withheld: the numbers are shown, but Mitzu won't say who won.
What it means for you: a withheld verdict is a problem in the test setup, not in the analysis. Fix the assignment in your experimentation tool. See Health checks.
7. Testing more things can be corrected for
Comparing several variants on several metrics gives chance more opportunities to produce a "significant" difference. Turn on the multiple-comparison correction and Mitzu applies a Holm correction across every variant on the primary and secondary metrics, so testing more doesn't raise the chance of a false win. It is off by default. Guardrails are never corrected, so they stay quick to flag harm.
What it means for you: with several variants or secondary metrics, turn the correction on before the test starts. Adding metrics is then not free: each one widens the intervals a little. Keep secondary metrics to the ones you would actually look at. See Statistics settings.
8. Results are computed when you ask, and once when the test ends
Nothing runs when you save or when you schedule the experiment. The warehouse is queried when you press Evaluate or Re-evaluate, and once on its own after the experiment end date has passed, to compute the final results and email them to the owner. Every page afterwards reads the stored results, so opening an experiment is instant and costs nothing.
What it means for you: evaluate when you want fresh numbers during the test. The final evaluation after the end date happens for you; re-evaluate yourself only if you need it sooner, or after a long conversion window. See Evaluating an experiment.
9. A decision is recorded against the results it was based on
Finalizing stores the outcome, the decision and your summary together with a frozen copy of the results. Later evaluations never overwrite them. Re-evaluating a finalized experiment compares today's numbers with the recorded decision; reopening it is the only way to adopt new results.
What it means for you: the record of what you decided and why stays true, even when the data or the metric definitions change afterwards. See Finalizing and Reopening and re-evaluating.
10. Experiments use the same definitions as the rest of Mitzu
Metrics are saved insights; the exposure event and variant property come from your event catalog; results become cohorts and annotations that every insight and dashboard can use.
What it means for you: when a saved insight is fixed, every experiment that measures it shows a note that its results are out of date until it is re-evaluated, so a metric never means two different things.