Limitations
Experiments is designed for the common case: a randomized test with a clear exposure event, 2 to 10 variants and a primary metric that is a rate or a per-user value. This page lists where the statistics are approximate and what isn't supported yet.
Statistical limits
Skewed value metrics with few buyers. Per-user and ratio metrics use the normal approximation, as most experimentation tools do by default. On very skewed metrics — revenue where a handful of customers spend a hundred times the median — with few buyers per variant, it is optimistic: a test of identical variants comes out "significant" more often than the confidence level promises. In simulation, with about 50 buyers per variant and heavy-tailed revenue, a nominal 5% false-win rate rose to about 12%; with 1,000 buyers per variant it was close to 6%.
Work around it: prefer a conversion metric ("share of exposed users who purchased") as the primary metric, keep revenue per user as a secondary, and plan for several hundred buyers per variant. Outlier capping is not available yet.
Fixed-horizon tests. The tests assume the result is read once, at the end of the test window. Mitzu enforces that by withholding the verdict until then, but nothing stops a decision taken on the running numbers. There are no sequential or always-valid tests.
Users in several variants are counted in each. They are not removed. While their extra counts stay under 5% of all counts this dilutes the difference slightly; above it the verdict is withheld.
Events at the exposure timestamp. An event recorded at exactly the same timestamp as the user's first exposure is not counted as after it.
Interactions between tests. Users in two tests at once are analysed in each independently. Mitzu doesn't check whether one test affected the other.
No variance reduction. Results don't use pre-experiment data (CUPED or similar) to narrow the intervals.
Not supported yet
| What | Instead |
|---|---|
| Results by segment (by platform, country, plan) | Save the variant cohorts after finalizing and break insights down by them; treat findings as ideas for the next test |
| Excluding cohorts such as internal users or bots | Add "Only where" filters on the exposure event, or exclude them in the saved insights' filters |
| Percentile metrics (p90 load time) | Test the share of users above a threshold, as a unique-users segmentation with a filter |
| Averages of a property, averages per active user | Use a sum per exposed user, or a ratio A / B |
| Retention and journey metrics | Use a funnel to the retained action within a conversion window |
| Funnels measuring time to convert | Test the conversion rate within the time you care about |
| More than 10 variants | Split the test, or narrow the exposure with "Only where" |
Practical limits
| Limit | Value |
|---|---|
| Variants per experiment | 2 to 10 |
| Primary metrics | exactly 1 |
| Minimum per variant for a comparison | 100 users and 5 conversions |
| Users in a frozen variant cohort | 200,000 |
| Experiment name | 120 characters |
| Variant display name | 80 characters |
| Confidence levels | 90%, 95%, 99% |