Claim and evidence
If the holdout group's engagement is higher, you've shipped yourself backward.Spotify (platform glossary) · confidence.spotify.com · 2026-07-03
Vendor claim
A tool vendor reports numbers about its own customers or product. Useful, but not independent.
Working · 2026-11 Statsig, GrowthBook and Spotify Confidence all document holdouts; benefit is largest for logged-in products.
What the numbers show
of users typically withheld from new features in a holdout (Statsig documentation guidance)
docs.statsig.comWhy it works
Single experiments measure one change; a holdout reveals the total effect and can show that individually positive changes hurt together.
How to run it
- Reserve 1 to 2% of users before launches
- Keep them out of the covered features
- Keep feature flags in place
- Compare against exposed users after three to six months
- Release the holdout and report the cumulative effect
Ways to run it
Design options to test, not promised results.
- Option AQuarterly or half-year holdout
Hold back a fresh group for a fixed period, report the cumulative effect, then release it and start a new one.
- Option BTeam-level holdout
Give each product area its own holdout so it can see the combined effect of its launches, as Spotify's glossary describes.
- Option CEngagement watch
Track whether the holdout's engagement ends up higher, a sign that positive tests added up to a negative effect.
Dated examples
- 2026
Spotify (platform glossary)
Glossary notes that holdouts have shown positive experiments adding up to negative cumulative effects
“If the holdout group's engagement is higher, you've shipped yourself backward.”
✓ quote checked word-for-word on the pageconfidence.spotify.com · 2026-07-03
Does it still work?
Statsig, GrowthBook and Spotify Confidence all document holdouts; benefit is largest for logged-in products. source
Risks
- Some users miss improvements
- Flag maintenance overhead
- Must be set up before the launches it should measure
Sources
- docs.statsig.com
“A holdout withholds a small percentage of users, typically between 1% and 2%, from a set of new features.”
✓ quote checked word-for-word on the page - docs.statsig.com
“Statsig recommends running a holdout for three to six months, then releasing it.”
✓ quote checked word-for-word on the page - docs.growthbook.io
“Holdouts require some users to miss out on improvements and new features while serving as the control group.”
✓ quote checked word-for-word on the page - confidence.spotify.com
Change log
- 2026-11-20 — harvest v0.1 (ops-ai)