Statistical methods

This page is for analysts and data scientists who want to know exactly how an experiment's results are computed. Experiments uses frequentist, fixed-horizon tests: each comparison is a test of one treatment against the control at the end of the test window, with a confidence interval for the relative lift.

What is read from the warehouse

An evaluation runs, per metric, an aggregate query over the exposed users of each variant. Only aggregates leave the warehouse; nothing per user does.

Metric kind	Per variant
Conversion (funnel, unique users)	n exposed users, x users who converted
Per-user value (event count, sum)	n exposed users, users with a value, Σx, Σx²
Ratio (A / B)	n exposed users, users with a value, Σx, Σx², Σy, Σy², Σxy, where x and y are each user's A and B

Each exposed user's values are summed over what they did after their first exposure in the window. A per-user mean is the ratio with y = 1 for every user. The same counts are also stored per day of first exposure, for the view over time, and kept only if they add up exactly to the totals.

Conversion metrics

Estimate. The relative lift of treatment v over control c is p̂_v / p̂_c − 1, with p̂ = x / n.

Test. A score test for the ratio of two independent proportions (the restricted maximum-likelihood score statistic of Miettinen and Nurminen, without the small-sample variance factor). Under the null ratio θ, the restricted estimates p̃_c and p̃_v = θ·p̃_c give

z(θ) = (p̂_v − θ·p̂_c) / √( p̃_v(1 − p̃_v)/n_v + θ²·p̃_c(1 − p̃_c)/n_c )

The p-value tests θ = 1: two-sided 2·(1 − Φ(|z|)), or one tail for a one-sided test.

Interval. The confidence interval for the ratio is the set of θ the score test doesn't reject, found by inverting z(θ) numerically. Subtracting 1 gives the interval for the relative lift. Score intervals keep good coverage at small rates, where a Wald interval is too narrow, and are asymmetric around the estimate.

Per-user and ratio metrics

Estimate. For each variant, the ratio of means r̂ = x̄ / ȳ, with its per-user variance by the delta method:

σ²_r = ( s²_x − 2·r̂·s_xy + r̂²·s²_y ) / ȳ²

The relative lift is r̂_v / r̂_c − 1, with standard error

SE = √( σ²_v / n_v + (r̂_v / r̂_c)²·σ²_c / n_c ) / r̂_c

where σ²_v and σ²_c are σ²_r of the treatment and of the control.

Test and interval. A z-test on the relative lift and the normal interval lift ± z·SE. The control's value must be positive for a relative lift to exist.

Minimum data

A metric is compared only when every variant has at least 100 exposed users and 5 conversions (or 5 users with a value). Below that the row reports Too few users for a comparison yet.

Multiple comparisons

With the correction on, all comparisons of treatments against the control on the primary and secondary metrics form one family, of size at most (variants − 1) × (primary + secondary metrics); comparisons that couldn't be made (not enough data, a failed query) are left out. Raw p-values are computed per comparison and adjusted with the Holm step-down procedure:

1. Sort the m p-values in ascending order.
2. The k-th smallest is compared with α / (m − k + 1); testing stops at the first that isn't below its level.
3. Adjusted p-values are max over j ≤ k of (m − j + 1)·p_(j), capped at 1.

The p-value shown is the adjusted one, and each interval is computed at the comparison's Holm level, so "the interval excludes zero" and "significant after correction" always agree.

Guardrails are evaluated at the unadjusted level, so they stay sensitive to harm. With the correction off, every comparison uses the unadjusted level, including between the variants of one metric.

One-sided tests

A one-sided test at level α uses the alternative in the metric's goal direction — or, on a guardrail, in the direction of harm. Its p-value is one tail of z. The reported bound is the one-sided bound, equal to the matching end of the two-sided interval at level 2α.

Significance and outcomes

A comparison is significant when its interval excludes zero (one-sided: when the tested bound is past zero). Its outcome is:

Role	Significant in the goal direction	Significant against it	Not significant
Primary, secondary	Significant win	Significant loss	Not significant
Guardrail	Within guardrail	Guardrail breached	Within guardrail

The verdict is taken from the primary metric: if any treatment wins, the one with the largest lift in the goal direction; otherwise, if any loses, the one with the largest loss; otherwise no clear winner. It is then overridden by, in order: a failed assignment health check (Verdict withheld) and, for a non-significant result or too few users, a sample plan not yet reached (Inconclusive: sample too small).

Sample plan

The required sample per variant is computed for 80% power at the experiment's confidence level and sidedness, Bonferroni-sized for the m comparisons the correction covers (m = 1 with the correction off):

z_α = Φ⁻¹(1 − α / (tails · m)),   z_β = Φ⁻¹(0.8)

• Conversion metrics, with baseline rate p1 and target p2 = p1·(1 + lift):

  n = (z_α + z_β)² · ( p1(1 − p1) + p2(1 − p2) ) / (p2 − p1)²

• Per-user and ratio metrics, with the control's observed value μ and per-user variance σ²:

  n = (z_α + z_β)² · 2σ² / (μ · lift)²

The plan is compared with the smallest variant. The remaining-days estimate divides the users still needed by the smallest variant's average daily intake so far.

Health-check thresholds

Check	Test	Threshold
Traffic split	Chi-square goodness of fit of users per variant against an even or the planned split	Fails or warns at p < 0.001
Users in several variants	Extra counts from users counted in more than one variant (counts − distinct users), as a share of all counts, users left out included	Fails above 5%
Sample against plan	Smallest variant against the required sample	Passes at 100%
Window coverage	Days covered against days in the window	Passes at full coverage
Metric tracking	Every metric computed, with a conversion or value in every variant	Fails on a failed metric query, warns on a variant with none
Users left out	Exposed users with no variant value	Warns above zero

The traffic split check withholds the verdict when it fails against a planned split, when a planned variant has no users, or when it fails against an even split on an experiment with a sample plan.

Metric details over time

For each day of the window, the metric is evaluated on the users first exposed up to that day, with the stored statistics settings, the same multiple-comparison family and the final control. The last day therefore equals the result in the results table. Other metrics in the family that have no daily counts enter earlier days' corrections with their whole-window p-values.

Determinism

Results are stored with the counts and the analysis that produced them. Pages read the stored analysis and never re-query; an analysis stored by an older release is re-derived from the stored counts. The same counts and settings always give the same verdict.