Claim and evidence
Do not add up the lifts of all winning tests; the sum overstates the true effect, so measure cumulative impact with a holdout or an adjusted method.
No dated example yet. Treat this tactic as a hypothesis.
Vendor claim
A tool vendor reports numbers about its own customers or product. Useful, but not independent.
Working · 2026-11 Datadog describes its own product; applies to any experimentation program.
What the numbers show
No published number for this tactic has passed our word-for-word check yet, so there is no chart.
Why it works
Tests that pass significance are a selected sample, so their estimates skew high. A holdout is not selected on significance and avoids the bias.
How to run it
- List launched winners
- Avoid a naive sum of their measured lifts
- Use a holdout for a direct estimate when traffic allows
- Otherwise use an adjusted cumulative-impact method and note its assumptions
- Report ranges, not a single sum
Ways to run it
Design options to test, not promised results.
- Option AHoldout estimate
Keep a group away from all launches and measure the combined effect directly when traffic allows.
- Option BAdjusted method
Apply a statistical correction to the summed lifts and state its assumptions next to the number.
- Option CRange reporting
Report cumulative impact as a range in planning documents instead of a single added-up figure.
Dated examples
No dated example yet. Treat this tactic as a hypothesis.
Does it still work?
Datadog describes its own product; applies to any experimentation program. source
Risks
- Holdouts cost traffic and time
- Adjusted methods rely on assumptions
Sources
- datadoghq.com
“That naive sum almost always overstates the truth because of a statistical artifact called the winner’s curse.”
✓ quote checked word-for-word on the page - datadoghq.com
“A holdout avoids the winner’s curse because its estimate is not selected for statistical significance.”
✓ quote checked word-for-word on the page
Change log
- 2026-11-20 — harvest v0.1 (ops-ai)