Skip to main content

Data requirements

Mitzu reads a test from the data your experimentation tool or your product already writes. Before you create an experiment, check that your warehouse has three things: an exposure event, a variant property on it, and the events you want to measure, recorded for the same users.

What Mitzu needs​

  1. An exposure event. An event in one of your event tables that is recorded when a user enters the test. Users without it in the test window are not part of the analysis.
  2. A variant property on the exposure event. An event property whose value says which variant the user saw: control, treatment, A, B, variant_2 and so on. Among the users exposed in the test window it must have between 2 and 10 values.
  3. Outcome events for the same users. The events your metrics count — signups, purchases, documents created — recorded with the same user identifier as the exposure event, so Mitzu can follow each exposed user to what they did next.
  4. Saved insights for the metrics. Each metric is a saved funnel or segmentation insight. See Metrics for which ones can be tested.

You don't need a separate table, a sync from your experimentation tool or any SDK.

Common shapes of exposure data​

A dedicated exposure event​

Most experimentation tools and customer data platforms record an event each time a user is shown a variant, often called Experiment Viewed, experiment_exposure or $exposure, with properties such as experiment_id, experiment_name and variation_id or variant.

  • Exposure event: Experiment Viewed
  • Only where: experiment_id is checkout-redesign, so the experiment only sees this test
  • Variant property: variation_id

One event usually carries every test you run, so the "Only where" filter is what keeps the tests apart. See Population and variants.

A feature flag evaluation event​

Feature-flag tools can log an event each time a flag is evaluated for a user, with the flag key and the value served.

  • Exposure event: the flag evaluation event, for example feature_flag_called
  • Only where: flag_key is new-onboarding
  • Variant property: the served value, for example variation

A product event that carries the variant​

Some teams add the variant as a property on an ordinary product event, such as a page view on the page being tested.

  • Exposure event: page_visit
  • Only where: page is /pricing, if the event fires on many pages
  • Variant property: experiment_variant

This works as long as the event fires when the user actually sees the tested experience.

Getting the data right​

Log exposure when the user experiences the variant, not when they are assigned. If every visitor is assigned at sign-in but only a few ever open the page being tested, an exposure logged at assignment adds thousands of users the change could not have affected. The real effect gets diluted and becomes much harder to detect.

Use the same user identifier everywhere. If the exposure event records an anonymous device ID and purchases record a logged-in user ID, Mitzu can't connect them, and the purchase rate per variant will look like zero. Make sure your event tables map both to the same user, or measure outcomes that are recorded with the same identifier.

Give each user one variant. A user who sees the control on one visit and the treatment on the next belongs to neither group. Mitzu counts such users in every variant they saw and, when those extra counts are more than 5% of all counts, withholds the verdict. See Health checks.

Keep variant values stable. Renaming a variant halfway through a test (from B to new_checkout) splits it into two variants in the data. Use display names in Mitzu instead; see Display names.

Checking your data before you create an experiment​

  1. Open a new segmentation insight in Mitzu.
  2. Pick the exposure event, add the "Only where" filters you plan to use, and break it down by the variant property.
  3. Set the date range to the test window and count unique users.

You should see one bar per variant, roughly in the proportions the test was set up with. The experiment form draws the same chart for you, under Exposed users per variant, and Review definition opens it as an insight.

tip

If the chart shows more than 10 values, the property is probably not the variant property — or the event carries several tests and needs an "Only where" filter.