Skip to main content

Why Experiments exists

An A/B test has two halves. One tool decides who sees which variant. The data that says whether the change worked — signups, purchases, documents created, cancellations — usually lives somewhere else: in your data warehouse, next to everything else your company knows about its customers.

Experiments joins the two halves. It takes the exposure data your experimentation or feature-flag tool (for example Optimizely, LaunchDarkly, GrowthBook or Statsig) or your own code writes to the warehouse, and measures the test with the saved insights your team already uses for everything else.

The problem it solves​

The tool that runs the test sees only part of the picture. An experimentation tool measures the events you send to it. The metrics a business is run on — revenue per customer, trial-to-paid conversion, retention, support tickets — are built in the warehouse from many sources, and most of them never reach the testing tool.

Two definitions of the same metric disagree. When "conversion" is defined once in the testing tool and again on the company dashboard, the test can win in one place and the dashboard can stay flat in the other. Nobody trusts either number.

Every test becomes a one-off analysis. Without a shared place to analyse tests, an analyst writes new SQL for each one: who was exposed, which variant they saw, how many converted, a significance test. The work is slow, hard to review and hard to repeat.

Decisions are lost. Months later, nobody remembers why the old checkout was kept, which variant won the pricing test, or whether the redesign was tested at all.

What Mitzu does differently​

  • Your metric definitions. An experiment metric is a saved insight — the same funnel or segmentation your dashboards show. Fix a definition once and the next evaluation of every experiment that uses it picks it up.
  • Your data, in place. Exposure, variants and outcomes are read from your warehouse with the same queries as the rest of Mitzu. Nothing is copied out, and no SDK is added to your product.
  • A verdict you can trust. Each result comes with a significance test, a confidence interval, an optional correction for testing many metrics at once and health checks that withhold the verdict when the test itself is broken.
  • A record of every decision. Finalizing an experiment stores the outcome, the decision, a written summary and the exact results it was based on. Anyone can look it up later.
  • The test feeds the rest of your analysis. Each variant's users can be saved as a cohort, and the test window and the roll-out can be marked on every chart as annotations.

What Mitzu does not do​

Experiments analyses tests; it does not run them.

  • It does not assign users to variants or decide who is in a test.
  • It does not create, change or switch off feature flags.
  • It does not stop a test early or change the traffic split.

Keep using your experimentation tool, or your own code, for all of that. Mitzu needs only the record each one leaves behind: an event that says a user was exposed to the test, with a property that says which variant they saw. See Data requirements.

Who it is for​

  • Product managers and marketers who run tests and need a clear answer and a place to record the decision. See For marketers and product managers.
  • Analysts and data teams who want every test measured the same way, on the same metric definitions, without writing SQL for each one.
  • Leadership who want to see what was tested, what won and what was rolled out.