Working · 2026-11Vendor claimGrowth opsAnalytics

Discount summed A/B winners (winner's curse)

Do not add up the lifts of all winning tests; the sum overstates the true effect, so measure cumulative impact with a holdout or an adjusted method.

Also called: cumulative impact estimation

Claim and evidence

The claimDo not add up the lifts of all winning tests; the sum overstates the true effect, so measure cumulative impact with a holdout or an adjusted method.

No dated example yet. Treat this tactic as a hypothesis.

What the numbers show

No published number for this tactic has passed our word-for-word check yet, so there is no chart.

Why it works

Diagram, not data
List launched testwinnersHoldout or adjustedestimateNaive sum of measuredliftsReport a range, notone sum
List launched test winnersHoldout oradjustedestimateNaive sum ofmeasured liftsReport a range, not one sum

Tests that pass significance are a selected sample, so their estimates skew high. A holdout is not selected on significance and avoids the bias.

How to run it

  1. List launched winners
  2. Avoid a naive sum of their measured lifts
  3. Use a holdout for a direct estimate when traffic allows
  4. Otherwise use an adjusted cumulative-impact method and note its assumptions
  5. Report ranges, not a single sum

Ways to run it

Design options to test, not promised results.

  1. Option AHoldout estimate

    Keep a group away from all launches and measure the combined effect directly when traffic allows.

  2. Option BAdjusted method

    Apply a statistical correction to the summed lifts and state its assumptions next to the number.

  3. Option CRange reporting

    Report cumulative impact as a range in planning documents instead of a single added-up figure.

Dated examples

No dated example yet. Treat this tactic as a hypothesis.

Does it still work?

Workingas of 2026-11

Datadog describes its own product; applies to any experimentation program. source

Risks

  • Holdouts cost traffic and time
  • Adjusted methods rely on assumptions

Sources

  1. datadoghq.com · Datadog blog · Lukas Goetz-Weiss and Eddie Cai · 2026-08-18
    “That naive sum almost always overstates the truth because of a statistical artifact called the winner’s curse.”
    ✓ quote checked word-for-word on the page
  2. datadoghq.com · Datadog blog · Lukas Goetz-Weiss and Eddie Cai · 2026-08-18
    “A holdout avoids the winner’s curse because its estimate is not selected for statistical significance.”
    ✓ quote checked word-for-word on the page

Change log

  • 2026-11-20 — harvest v0.1 (ops-ai)