Claim and evidence
Review known experiment pitfalls (Simpson's paradox, short-term versus long-term effects) before declaring a winner.
No dated example yet. Treat this tactic as a hypothesis.
Primary
The company or founder describes the tactic and its result themselves: own blog, talk, filing.
Working · 2026-11 The ExP page lists the papers; the papers themselves were not fetched.
What the numbers show
No published number for this tactic has passed our word-for-word check yet, so there is no chart.
Why it works
Uneven traffic allocation across periods can flip the apparent winner, and short-term metrics may not predict the long term. A standing pitfall checklist prevents repeating these mistakes.
How to run it
- Keep allocation constant during the test or analyze by period
- Segment results to check for reversals
- Use long-term metrics or holdouts for big changes
- Re-run surprising results
- Keep a pitfall checklist
Ways to run it
Design options to test, not promised results.
- Option AConstant allocation
Keep the traffic split fixed for the whole test, ramping before measurement starts.
- Option BPeriod-by-period analysis
If the split must change, compare variants within each period instead of pooling all days.
- Option CPitfall checklist
Review a standing list of known traps, including short-term versus long-term effects, before declaring a winner.
Timeline
- 12009 first documented: Crook, Frasca, Kohavi and Longbotham, KDD 2009 (listed on Microsoft ExP site)
- 22026-11 status checked: Working
Dated examples
No dated example yet. Treat this tactic as a hypothesis.
Does it still work?
The ExP page lists the papers; the papers themselves were not fetched. source
Risks
- Only paper titles and excerpts were verified
- Easy to over-segment and find false patterns
Sources
- exp-platform.com
“it's easy to incorrectly identify the winning Treatment because of Simpson's paradox.”
○ page blocks automated checks; quote as relayed - exp-platform.com
“The difficulty is that short-term changes to metrics may not predict the long-term impact of a change.”
○ page blocks automated checks; quote as relayed
Change log
- 2026-11-20 — harvest v0.1 (ops-ai)