Claim and evidence
Decide sample size before an A/B test and avoid stopping at the first significant result.
No dated example yet. Treat this tactic as a hypothesis.
Secondary
A reputable third party analysed it (Lenny's Newsletter, First Round Review, Reforge, an Ahrefs study).
Working · 2026-11 Classic 2010 essay; modern tools add sequential tests so peeking is safer (see sequential-testing).
What the numbers show
No published number for this tactic has passed our word-for-word check yet, so there is no chart.
Why it works
Checking repeatedly and stopping when the result looks significant inflates false positives. Committing to a sample size in advance removes the problem.
How to run it
- Compute required sample size before launch
- Write down the stopping rule
- Hide or ignore interim significance
- Read results only at the planned end
- Use sequential methods if early stopping is required
Ways to run it
Design options to test, not promised results.
- Option APre-registered sample size
Compute the required sample before launch and write down the stopping rule.
- Option BHidden interim results
Hide significance from the dashboard until the planned end so no one is tempted to stop early.
- Option CSequential testing
If early stopping is a must, use a sequential method designed for repeated looks.
Timeline
- 12010 first documented: Evan Miller, 'How Not To Run An A/B Test'
- 22026-11 status checked: Working
Dated examples
No dated example yet. Treat this tactic as a hypothesis.
Does it still work?
Classic 2010 essay; modern tools add sequential tests so peeking is safer (see sequential-testing). source
Risks
- Business pressure to call winners early
- Underpowered tests waste traffic
Sources
- evanmiller.org
“The more you peek, the more your significance levels will be off.”
✓ quote checked word-for-word on the page - evanmiller.org
“Committing to a sample size completely mitigates the problem described here.”
✓ quote checked word-for-word on the page
Change log
- 2026-11-20 — harvest v0.1 (ops-ai)