Claim and evidence
If the test group has more crashes, this would result in more users in the test groupStatsig (illustration) · statsig.com · 2023-08-31
Vendor claim
A tool vendor reports numbers about its own customers or product. Useful, but not independent.
Working · 2026-11 Built into major platforms; Optimizely runs a sequential SRM check throughout a test.
What the numbers show
No published number for this tactic has passed our word-for-word check yet, so there is no chart.
Why it works
An unexpected split signals a bug in assignment, logging or exposure that biases the result. Platforms run a chi-squared or sequential test and flag problems early.
How to run it
- Set the planned allocation
- Run an SRM test on exposure counts
- Investigate flagged tests by segment (browser, region, SDK)
- Fix the cause and restart
- Never ship on a test with unexplained SRM
Ways to run it
Design options to test, not promised results.
- Option AAutomatic platform check
Rely on a testing platform that runs the mismatch test continuously and flags it on the results page.
- Option BSegment drill-down
When flagged, split counts by browser, region and app version to find where users go missing.
- Option CHard ship rule
Write into the test process that no result with an unexplained mismatch can be shipped.
Timeline
- 12023 Statsig (illustration) — Vendor blog lists causes such as crashes, bot filtering and duplicated data that skew group sizes
- 22026-11 status checked: Working
Dated examples
- 2023
Statsig (illustration)
Vendor blog lists causes such as crashes, bot filtering and duplicated data that skew group sizes
“If the test group has more crashes, this would result in more users in the test group”
✓ quote checked word-for-word on the pagestatsig.com · 2023-08-31
Does it still work?
Built into major platforms; Optimizely runs a sequential SRM check throughout a test. source
Risks
- Checks flag problems but do not fix them
- Chi-squared on huge samples flags tiny harmless gaps
Sources
- docs.statsig.com
“the observed allocation of unique users between test groups differs from the expected allocation”
✓ quote checked word-for-word on the page - docs.optimizely.com
“Optimizely's SSRM algorithm continuously checks traffic counts throughout an A/B experiment.”
✓ quote checked word-for-word on the page - statsig.com
Change log
- 2026-11-20 — harvest v0.1 (ops-ai)