Working · 2026-11Vendor claimGrowth opsAnalytics

Long-running product holdout groups

Keep a small share of users from new features for months to measure their combined effect.

Also called: holdout, universal holdout

Claim and evidence

The claimIf the holdout group's engagement is higher, you've shipped yourself backward.

Spotify (platform glossary) · confidence.spotify.com · 2026-07-03

What the numbers show

1%–2%

of users typically withheld from new features in a holdout (Statsig documentation guidance)

docs.statsig.com

Why it works

Diagram, not data
Reserve asmall share ofusersShip featuresto everyoneelseExposed usersget everylaunchHoldout keepsthe oldproductCompare aftermonths, thenrelease
Reserve a small share of usersShip features to everyone elseExposed usersget everylaunchHoldout keepsthe old productCompare after months, thenrelease

Single experiments measure one change; a holdout reveals the total effect and can show that individually positive changes hurt together.

How to run it

  1. Reserve 1 to 2% of users before launches
  2. Keep them out of the covered features
  3. Keep feature flags in place
  4. Compare against exposed users after three to six months
  5. Release the holdout and report the cumulative effect

Ways to run it

Design options to test, not promised results.

  1. Option AQuarterly or half-year holdout

    Hold back a fresh group for a fixed period, report the cumulative effect, then release it and start a new one.

  2. Option BTeam-level holdout

    Give each product area its own holdout so it can see the combined effect of its launches, as Spotify's glossary describes.

  3. Option CEngagement watch

    Track whether the holdout's engagement ends up higher, a sign that positive tests added up to a negative effect.

Dated examples

  1. 2026

    Spotify (platform glossary)

    Glossary notes that holdouts have shown positive experiments adding up to negative cumulative effects

    “If the holdout group's engagement is higher, you've shipped yourself backward.”
    ✓ quote checked word-for-word on the page

    confidence.spotify.com · 2026-07-03

Does it still work?

Workingas of 2026-11

Statsig, GrowthBook and Spotify Confidence all document holdouts; benefit is largest for logged-in products. source

Risks

  • Some users miss improvements
  • Flag maintenance overhead
  • Must be set up before the launches it should measure

Sources

  1. docs.statsig.com · Statsig docs · 2025-09-18
    “A holdout withholds a small percentage of users, typically between 1% and 2%, from a set of new features.”
    ✓ quote checked word-for-word on the page
  2. docs.statsig.com · Statsig docs · 2025-09-18
    “Statsig recommends running a holdout for three to six months, then releasing it.”
    ✓ quote checked word-for-word on the page
  3. docs.growthbook.io · GrowthBook docs · undated
    “Holdouts require some users to miss out on improvements and new features while serving as the control group.”
    ✓ quote checked word-for-word on the page
  4. confidence.spotify.com · Spotify Confidence glossary · 2026-07-03

Change log

  • 2026-11-20 — harvest v0.1 (ops-ai)