Claim and evidence
Use a sequential test design so you can stop early on a clear winner without breaking statistical validity.
No dated example yet. Treat this tactic as a hypothesis.
Secondary
A reputable third party analysed it (Lenny's Newsletter, First Round Review, Reforge, an Ahrefs study).
Working · 2026-11 GrowthBook offers sequential testing to mitigate peeking; Bayesian engines can still inflate false positives if stopped early.
What the numbers show
No published number for this tactic has passed our word-for-word check yet, so there is no chart.
Why it works
Sequential rules set boundaries that account for repeated looks. The price is longer tests when there is no effect, in return for faster calls on large effects.
How to run it
- Choose a sequential procedure or a platform that supports it
- Set the minimum detectable effect and stopping rule
- Run and monitor without ad-hoc stopping
- Stop only when a boundary is crossed
- Do not extend the test beyond the limit
Ways to run it
Design options to test, not promised results.
- Option APlatform sequential mode
Switch on the sequential option in your testing tool so peeking at results does not inflate false positives.
- Option BPre-set stopping rule
Write the stopping boundary and maximum length into the plan before launch.
- Option CFixed-horizon for small effects
Keep a classic fixed-length test when you expect a small effect and can wait.
Timeline
- 12015 first documented: Evan Miller, 'Simple Sequential A/B Testing'
- 22026-11 status checked: Working
Dated examples
No dated example yet. Treat this tactic as a hypothesis.
Does it still work?
GrowthBook offers sequential testing to mitigate peeking; Bayesian engines can still inflate false positives if stopped early. source
Risks
- No-effect tests run longer than fixed-sample designs
- Must follow the stopping rule exactly
Sources
- evanmiller.org
“Stopping an A/B test early because the results are statistically significant is usually a bad idea.”
✓ quote checked word-for-word on the page - docs.growthbook.io
“you can enable sequential testing to mitigate concerns with peeking.”
✓ quote checked word-for-word on the page
Change log
- 2026-11-20 — harvest v0.1 (ops-ai)