Glossary

The terms below appear in the experiment pages and throughout this documentation. Each links to the page that explains it in full.

A/A test — A test where every variant is identical. Useful for checking the setup: an A/A test should show No clear winner about 95 times out of 100 at 95% confidence.

Allocation window — The start date and the allocation end date, both included. Only users exposed in it are enrolled in the experiment. See Test window and scheduling.

Baseline rate — The primary metric's conversion rate without the change. Used by the sample plan; left empty, Mitzu uses the control's observed rate.

Confidence interval — The range of lifts that are consistent with the data at the chosen confidence level, for example 95% interval +2.1% to +9.8%. If the interval doesn't include 0%, the difference is significant. See Results and verdicts.

Confidence level — How strong the evidence must be before a difference counts: 90%, 95% (the default) or 99%. See Statistics settings.

Control — The variant every other variant is compared with — usually the experience users had before the test. See Population and variants.

Definition — Everything the results depend on: the population, metrics, statistics settings, sample plan, variants and test window. It is locked once the experiment is scheduled. See Editing and locking.

Evaluate — Query the warehouse and compute every metric for every variant over the test window so far. The only action that runs a query. See Evaluating an experiment.

Experiment end date — Later than the allocation end, so users enrolled late still have time to convert. The days from the allocation end to the end date set how long each user is measured after their first exposure. The final evaluation runs on its own after it. See Test window and scheduling.

Exposed users — The users who have the exposure event in the allocation window, counted per variant.

Exposure event — The event that marks a user entering the test, for example experiment_viewed or page_visit. Only users with this event in the test window are analysed. See Population and variants.

Finalize — Record the outcome, the decision and a summary, and freeze the results. See Finalizing.

Guardrail — A metric that must not get worse, such as cancellations or error rate. A guardrail is Within guardrail or Guardrail breached; it never wins. See Metrics.

Health checks — Checks run on every result that look for problems in the test itself: an uneven traffic split, users seen in several variants, too small a sample. See Health checks.

Holm correction — The multiple-comparison correction Mitzu applies, when you turn it on, across every variant on the primary and secondary metrics. Off by default. See Statistical methods.

Hypothesis — What you changed and what you expect it to do. Required on every experiment.

Inconclusive: sample too small — No clear winner, or too few users yet, and the smallest variant stopped short of the planned sample, so a real lift could have been missed. Called Underpowered before October 2026. See Results and verdicts.

Lift — The relative difference between a variant and the control. A variant converting at 5.5% against a control at 5.0% has a lift of +10%.

Minimum lift / smallest lift worth detecting — The smallest relative change on the primary metric you would act on. With it, the sample plan says how many users each variant needs. See Sample plan.

No clear winner — The variants performed about the same as the control on the primary metric. Called No significant difference before October 2026. See Results and verdicts.

One-sided test — A test that only looks for the change you want — an increase on a metric that should go up, harm on a guardrail. See Statistics settings.

p-value — The probability of seeing a difference at least this large if the variants were really the same. Shown as p = 0.012 or p < 0.001.

Planned split — The share of users the test was set up to send to each variant. Leave it empty for an even split. See Population and variants.

Population — Who is in the test: the exposure event, its "Only where" filters and the variant property.

Primary metric — The one metric that decides the test. Every experiment has exactly one.

Secondary metric — A metric that adds supporting evidence but doesn't decide the test.

Significant — A difference is significant when its confidence interval excludes zero after any correction: it is unlikely to be chance alone.

Significant win badge — Significant win: yes or Significant win: no under the verdict headline. Not shown while the test is running, with too few users to compare, or while the verdict is withheld. See Results and verdicts.

Test window — From the start date to the experiment end date. It is split in two: the allocation window (start date to allocation end, both included), in which exposed users are enrolled, and the days up to the experiment end date, which set how long their metrics are measured. See Test window and scheduling.

Too few users to compare — A variant has too few users or conversions for any comparison. Called Not enough data before October 2026. See Results and verdicts.

Variant — One version of the experience being tested, including the control. An experiment compares 2 to 10 variants.

Variant cohort — A cohort holding the users exposed to one variant, saved from a finalized experiment. See Cohorts, annotations and follow-up tests.

Variant property — The event property on the exposure event that says which variant a user saw, for example variant or experiment_variant.

Verdict — The headline of the Results card: whether a variant beats or loses to the control on the primary metric. See Results and verdicts.

Verdict withheld — The verdict Mitzu shows when a health check finds the comparison untrustworthy. See Health checks.