Scenarios and use cases
The examples below cover the tests marketing and product teams run most often. Each one shows how to set the experiment up in Mitzu, what you are likely to see, and how to decide. The event and property names are examples; use the ones in your own data.
Landing page headline
The test. Two new headlines on the paid-search landing page against the current one.
| Setting | Value |
|---|---|
| Exposure event | page_visit |
| Only where | page is /lp/analytics |
| Variant property | experiment_variant: Control, A, B (display names Current, Outcome-led, Question) |
| Primary | Visitor to signup funnel, 1-day conversion window, increase is good |
| Secondary | Visitor to pricing page funnel |
| Guardrail | Visitor to trial started funnel, should not decrease: more signups shouldn't mean worse ones |
| Sample plan | Smallest lift worth detecting 10% |
| Test window | Two full weeks |
What to watch. Landing pages get users from campaigns that start and stop. If a campaign sends traffic only to one variant's URL, the split breaks and the Traffic split check fails. Make sure assignment happens on the page, after the click.
Deciding. With three variants, turn on the multiple-comparison correction: it covers two
treatments compared against the control. If
Outcome-led wins and Question doesn't, roll out Outcome-led. If both win, the verdict names the
larger lift; check whether their intervals overlap before claiming one is better than the other — the
test compared each with the control, not with each other.
Pricing page: annual plan first
The test. Showing the annual plan first on the pricing page, against monthly first.
| Setting | Value |
|---|---|
| Exposure event | pricing_page_viewed |
| Variant property | variant: control, annual_first |
| Primary | Share of exposed users who bought an annual plan (segmentation of subscription_started where interval is annual, unique users) |
| Secondary | Revenue per exposed user (segmentation summing amount on subscription_started) |
| Guardrail | Share of exposed users who bought any plan, should not decrease |
| Statistics | 99% confidence: pricing changes are expensive to reverse |
What to watch. The guardrail is the point of this test: shifting buyers to annual is only good if
it doesn't lose buyers overall. A significant win on the primary with Guardrail breached on the
overall purchase rate means the page sells more annual plans to fewer people.
Deciding. Revenue per exposed user is the tiebreaker. A secondary metric doesn't decide the test, but a win on the primary, a guardrail within limits and revenue per user up or flat makes a clear case to roll out.
Onboarding checklist
The test. A checklist on the first screen after signup, against no checklist.
| Setting | Value |
|---|---|
| Exposure event | onboarding_started |
| Variant property | checklist_variant |
| Primary | Signup to first document created funnel, 7-day conversion window |
| Secondary | Events per exposed user: document_created |
| Guardrail | Support tickets per exposed user, should not increase |
| Test window | Three weeks, plus 7 days before the final evaluation |
What to watch. Onboarding changes often look great in the first days and fade: curious users click everything new. Open Show details on the primary metric and check whether the lift settled. See Metric details over time.
While it runs. The experiment shows Incomplete: no verdict yet. Check health weekly, don't
decide.
Trial reminder email timing
The test. Sending the trial-ending reminder on day 10 instead of day 12 of a 14-day trial.
| Setting | Value |
|---|---|
| Exposure event | trial_reminder_assigned (logged by the lifecycle tool when the user enters the test) |
| Variant property | send_day: 12 (control), 10 |
| Primary | Trial to paid funnel, 14-day conversion window |
| Guardrail | Unsubscribe rate, should not increase |
| Sample plan | Baseline rate 18%, smallest lift worth detecting 8% |
What to watch. Email tests reach fewer users than page tests, so they often end Inconclusive: sample too small: no clear winner, and fewer users than the plan needed. That is not "the timing doesn't matter"; it is "this test couldn't tell".
Deciding. Finalize as Inconclusive with Re-test, then Duplicate as follow-up with a longer
window — or decide the smaller effect isn't worth chasing and record Stop.
Checkout redesign with three variants
The test. Two redesigns of the checkout page against the current one, at 34/33/33.
| Setting | Value |
|---|---|
| Exposure event | checkout_viewed |
| Variant property | variant: control, single_page, express |
| Planned split | 34% / 33% / 33% |
| Primary | Checkout to order completed funnel, 1-day conversion window |
| Secondary | Average order value: segmentation with sum of revenue on order_completed as A, order_completed events as B, formula A / B |
| Guardrail | Payment errors per exposed user, should not increase |
What to watch. With two treatments and a secondary metric, turn on the multiple-comparison
correction; it covers four comparisons. The settings line under the results says
95% confidence · two-sided test · Holm correction across 4 comparisons.
Deciding. A test can have a winner and a loser at once: single_page wins, express loses. The
verdict names the winner; the loser's row is the reason not to ship express. Record both in the
summary.
Feature roll-out as a "do no harm" test
The test. A rebuilt search back end, rolled out to 10% of users behind a feature flag. Nobody expects users to search more; the question is whether anything got worse.
| Setting | Value |
|---|---|
| Exposure event | feature_flag_evaluated |
| Only where | flag_key is new-search |
| Variant property | served_value: off (control), on |
| Planned split | off 90%, on 10% |
| Primary | Searches leading to a result click (funnel), increase is good |
| Guardrails | Errors per exposed user and sessions per exposed user |
| Statistics | One-sided: the guardrails then look only for harm, which is the question here |
What to watch. Set the planned split, or the Traffic split check flags the 90/10 split as uneven.
Deciding. No clear winner on the primary with every guardrail Within guardrail is the
outcome you hoped for: finalize as No clear winner with Roll out, and set the roll-out date in the
Finalize modal.
A test with a broken assignment
The test. A homepage test where assignment was stored in a cookie that was reset on login.
What you see. Verdict withheld: users were seen in several variants. The
Users in several variants check fails, for
example with 16% of the counts coming from users who saw both versions.
Deciding. The numbers can't be rescued: the groups are mixed. Finalize as Inconclusive with
Re-test, explain the cookie problem in the summary, fix the assignment in your tool and run the test
again as a follow-up.
Analysing a test that already ended
The situation. A test ran last quarter and was analysed in a spreadsheet. You want it on record in Mitzu with the same rigour as new tests.
- Create the experiment with the test's real start and end dates.
- Set the population, metrics and statistics as they were planned at the time, not as the spreadsheet ended up.
- Schedule it. With a past window it goes straight to Running.
- Evaluate. The experiment becomes Ready for decision with a verdict.
- Finalize with the decision that was actually taken, and note in the summary that it was recorded afterwards.
If Mitzu's numbers differ from the spreadsheet, the usual reasons are that Mitzu counts only users after their first exposure, uses each insight's own conversion window, and counts users seen in several variants in each of them. See Reading results without statistics.