Working · 2026-11Vendor claimGrowth opsAnalytics

Sample ratio mismatch check

Automatically test that the observed split of users between variants matches the planned split, and distrust results when it does not.

Also called: SRM check

Claim and evidence

The claimIf the test group has more crashes, this would result in more users in the test group

Statsig (illustration) · statsig.com · 2023-08-31

What the numbers show

No published number for this tactic has passed our word-for-word check yet, so there is no chart.

Why it works

Diagram, not data
Planned split versusobserved countsSplit matches: readthe resultsMismatch: stop andinvestigateFix the cause andrestart
Planned split versus observedcountsSplit matches:read theresultsMismatch: stopand investigateFix the cause and restart

An unexpected split signals a bug in assignment, logging or exposure that biases the result. Platforms run a chi-squared or sequential test and flag problems early.

How to run it

  1. Set the planned allocation
  2. Run an SRM test on exposure counts
  3. Investigate flagged tests by segment (browser, region, SDK)
  4. Fix the cause and restart
  5. Never ship on a test with unexplained SRM

Ways to run it

Design options to test, not promised results.

  1. Option AAutomatic platform check

    Rely on a testing platform that runs the mismatch test continuously and flags it on the results page.

  2. Option BSegment drill-down

    When flagged, split counts by browser, region and app version to find where users go missing.

  3. Option CHard ship rule

    Write into the test process that no result with an unexplained mismatch can be shipped.

Timeline

  1. 12023 Statsig (illustration) — Vendor blog lists causes such as crashes, bot filtering and duplicated data that skew group sizes
  2. 22026-11 status checked: Working

Dated examples

  1. 2023

    Statsig (illustration)

    Vendor blog lists causes such as crashes, bot filtering and duplicated data that skew group sizes

    “If the test group has more crashes, this would result in more users in the test group”
    ✓ quote checked word-for-word on the page

    statsig.com · 2023-08-31

Does it still work?

Workingas of 2026-11

Built into major platforms; Optimizely runs a sequential SRM check throughout a test. source

Risks

  • Checks flag problems but do not fix them
  • Chi-squared on huge samples flags tiny harmless gaps

Sources

  1. docs.statsig.com · Statsig docs · undated
    “the observed allocation of unique users between test groups differs from the expected allocation”
    ✓ quote checked word-for-word on the page
  2. docs.optimizely.com · Optimizely docs · 2026-08-05
    “Optimizely's SSRM algorithm continuously checks traffic counts throughout an A/B experiment.”
    ✓ quote checked word-for-word on the page
  3. statsig.com · Statsig blog · Liz Obermaier · 2023-08-31

Change log

  • 2026-11-20 — harvest v0.1 (ops-ai)