Working · 2026-11SecondaryGrowth opsAnalytics

Sequential testing for early stopping

Use a sequential test design so you can stop early on a clear winner without breaking statistical validity.

Also called: always-valid inference, sequential A/B test

Claim and evidence

The claimUse a sequential test design so you can stop early on a clear winner without breaking statistical validity.

No dated example yet. Treat this tactic as a hypothesis.

What the numbers show

No published number for this tactic has passed our word-for-word check yet, so there is no chart.

Why it works

Diagram, not data
Evidence collected over timeKeep monitoringBoundary crossed: stopPlanned maximum lengthNo ad-hoc stopping
Evidence collected over timeKeep monitoringBoundary crossed: stopPlanned maximum lengthNo ad-hoc stopping

Sequential rules set boundaries that account for repeated looks. The price is longer tests when there is no effect, in return for faster calls on large effects.

How to run it

  1. Choose a sequential procedure or a platform that supports it
  2. Set the minimum detectable effect and stopping rule
  3. Run and monitor without ad-hoc stopping
  4. Stop only when a boundary is crossed
  5. Do not extend the test beyond the limit

Ways to run it

Design options to test, not promised results.

  1. Option APlatform sequential mode

    Switch on the sequential option in your testing tool so peeking at results does not inflate false positives.

  2. Option BPre-set stopping rule

    Write the stopping boundary and maximum length into the plan before launch.

  3. Option CFixed-horizon for small effects

    Keep a classic fixed-length test when you expect a small effect and can wait.

Timeline

  1. 12015 first documented: Evan Miller, 'Simple Sequential A/B Testing'
  2. 22026-11 status checked: Working

Dated examples

No dated example yet. Treat this tactic as a hypothesis.

Does it still work?

Workingas of 2026-11

GrowthBook offers sequential testing to mitigate peeking; Bayesian engines can still inflate false positives if stopped early. source

Risks

  • No-effect tests run longer than fixed-sample designs
  • Must follow the stopping rule exactly

Sources

  1. evanmiller.org · evanmiller.org · Evan Miller · 2015-10-13
    “Stopping an A/B test early because the results are statistically significant is usually a bad idea.”
    ✓ quote checked word-for-word on the page
  2. docs.growthbook.io · GrowthBook docs · undated
    “you can enable sequential testing to mitigate concerns with peeking.”
    ✓ quote checked word-for-word on the page

Change log

  • 2026-11-20 — harvest v0.1 (ops-ai)