Working · 2026-11PrimaryGrowth opsAnalytics

Check for Simpson's paradox and short-term bias

Review known experiment pitfalls (Simpson's paradox, short-term versus long-term effects) before declaring a winner.

Also called: experiment pitfalls review

Claim and evidence

The claimReview known experiment pitfalls (Simpson's paradox, short-term versus long-term effects) before declaring a winner.

No dated example yet. Treat this tactic as a hypothesis.

What the numbers show

No published number for this tactic has passed our word-for-word check yet, so there is no chart.

Why it works

Diagram, not data
Traffic split changedmid-test?Analyse each periodseparatelyPooled totals canflip the winnerRe-run surprisingresults
Traffic split changed mid-test?Analyse eachperiodseparatelyPooled totalscan flip thewinnerRe-run surprising results

Uneven traffic allocation across periods can flip the apparent winner, and short-term metrics may not predict the long term. A standing pitfall checklist prevents repeating these mistakes.

How to run it

  1. Keep allocation constant during the test or analyze by period
  2. Segment results to check for reversals
  3. Use long-term metrics or holdouts for big changes
  4. Re-run surprising results
  5. Keep a pitfall checklist

Ways to run it

Design options to test, not promised results.

  1. Option AConstant allocation

    Keep the traffic split fixed for the whole test, ramping before measurement starts.

  2. Option BPeriod-by-period analysis

    If the split must change, compare variants within each period instead of pooling all days.

  3. Option CPitfall checklist

    Review a standing list of known traps, including short-term versus long-term effects, before declaring a winner.

Timeline

  1. 12009 first documented: Crook, Frasca, Kohavi and Longbotham, KDD 2009 (listed on Microsoft ExP site)
  2. 22026-11 status checked: Working

Dated examples

No dated example yet. Treat this tactic as a hypothesis.

Does it still work?

Workingas of 2026-11

The ExP page lists the papers; the papers themselves were not fetched. source

Risks

  • Only paper titles and excerpts were verified
  • Easy to over-segment and find false patterns

Sources

  1. exp-platform.com · Microsoft Experimentation Platform (ExP) site · undated
    “it's easy to incorrectly identify the winning Treatment because of Simpson's paradox.”
    ○ page blocks automated checks; quote as relayed
  2. exp-platform.com · Microsoft Experimentation Platform (ExP) site · undated
    “The difficulty is that short-term changes to metrics may not predict the long-term impact of a change.”
    ○ page blocks automated checks; quote as relayed

Change log

  • 2026-11-20 — harvest v0.1 (ops-ai)