Running experiments in Mitzu

This page is for the people who own the tests: product managers, growth and lifecycle marketers, designers. It describes a routine that works — what to do before, during and after a test, who does what, and where in Mitzu each step happens. You don't need a statistics background; Mitzu does the maths and tells you, in words, what the numbers support.

Who does what

Role	Responsibilities
Test owner (PM, marketer)	Writes the hypothesis, picks the primary metric and guardrails, creates the experiment, reads the result, finalizes it with the decision
Analyst or data person	Checks the population and the metric definitions once per new kind of test, reviews surprising results
Engineer / tool admin	Sets up the test in the experimentation tool and makes sure exposure is logged with the variant

In a small team one person does all three. The split still helps as a checklist.

Before the test

1. Write the hypothesis

Use one sentence with three parts: the change, the metric and the direction.

Showing the annual plan first on the pricing page will increase the share of pricing page visitors who buy an annual plan, without lowering the share who buy any plan.

A hypothesis that names no metric can't be wrong, so it can't be tested. Put it in the experiment's Hypothesis field; the finalize summary quotes it.

2. Pick the metrics

• One primary metric — the number you would act on. Closer to the change is better: a pricing page test is judged on pricing page purchases, not on yearly revenue.
• Guardrails — what must not get worse: cancellations, refunds, support tickets, unsubscribes, error rates.
• One or two secondary metrics — the next step in the funnel, a second outcome you expect to move.

Each metric is a saved insight. If the one you need doesn't exist, build it in Mitzu (a funnel or a segmentation), check it with your analyst and save it. See Metrics.

3. Size the test

Decide the smallest lift worth detecting: the change that would justify rolling the variant out. Enter it in the sample plan. Mitzu works out how many users each variant needs; divide by your daily traffic per variant to get the test length, and round up to whole weeks.

If the answer is "four months", don't run it as is. Test a bolder change, test on a page with more traffic, or pick a metric closer to the change.

4. Agree the decision rule

Write down, before the test starts, what each outcome will lead to: "If Annual first wins without a guardrail breach, we roll it out. If there is no difference, we keep the current page because it is simpler." This takes the pressure off the result and stops a debate after it.

5. Set it up

1. Configure the test in your experimentation tool. Note its key.
2. In Mitzu, click Create → Experiment. Fill in the details, the population, the test window, the statistics (keep the defaults), the sample plan and the metrics. Add the tool's key and a link.
3. Ask your analyst to check the population once. Review definition next to the exposed-users chart opens it as an insight they can inspect.
4. Click Schedule experiment when the test goes live. The definition is now locked.

While the test runs

Evaluate once or twice a week, not to decide, but to catch problems early:

• Is exposure coming in? The Exposed users row should grow every time.
• Is the split right? The Traffic split check should pass.
• Does anyone see two variants? Users in several variants should pass.
• Is every metric tracked? Metric tracking should pass; a variant with zero conversions is usually a tracking bug.

A failing assignment check is worth fixing immediately — every day it stays broken is a day of unusable data.

Don't decide on the numbers yet. The Results card says Incomplete: no verdict yet for a reason. See Why not stop a test early?

When the test ends

1. Stop or freeze the test in your tool as planned.
2. Wait for the conversion window. If the primary metric is "trial started within 7 days", users exposed on the last day need 7 more days.
3. Evaluate once more. The experiment becomes Ready for decision and shows a verdict.
4. Read the result. Check the verdict, the guardrails and the health checks. Open Show details on the primary metric to see whether the lift was stable or faded. See Reading results without statistics.
5. Finalize. Choose the outcome and decision — normally the ones agreed before the test — and edit the drafted summary. See Finalizing.
6. Follow up. Save the variant cohorts if you want to track the users later, and add the test window to the annotations.

After the decision

• Roll out in your tool, and record the date it went live as the Roll-out date when you finalize with Roll out. Mitzu adds the roll-out annotation for you.
• Share the result. Copy the summary from the Summary card into Slack, email or your product review. Link to the experiment for the details.
• Plan the next test. Duplicate as follow-up starts a new draft with the same setup.

A weekly rhythm for a testing programme

When	What
Monday	Open the experiments list sorted by Status. Evaluate every running test; check health
When a test ends	Final evaluation after the conversion window; finalize within the week
Monthly	Filter Ready for decision: nothing should wait there for long
Quarterly	Filter Finalized and a quarter tag. Review what was tested, what won and what shipped

Conventions that pay off:

• Name experiments <Area>: <change>, for example Pricing: annual plan first.
• Tag by product area and quarter.
• Always fill in the tool's key and link.
• Finalize every test, including the ones with no winner. A record of what didn't work saves the next person from testing it again.

What to expect

Most tests don't win. Across the industry, only a minority of A/B tests produce a significant improvement on their primary metric, and that is normal: each test that comes out flat is a change you didn't ship on a hunch. Judge a testing programme by how many decisions it informed, not by its win rate.