Working · 2026-11SecondaryGrowth opsAnalytics

Fixed sample size, no early stopping

Decide sample size before an A/B test and avoid stopping at the first significant result.

Also called: peeking problem

Claim and evidence

The claimDecide sample size before an A/B test and avoid stopping at the first significant result.

No dated example yet. Treat this tactic as a hypothesis.

What the numbers show

No published number for this tactic has passed our word-for-word check yet, so there is no chart.

Why it works

Diagram, not data
Test is runningRead only at theplanned sampleStop at firstsignificant peekFalse positives stayin check
Test is runningRead only atthe plannedsampleStop at firstsignificantpeekFalse positives stay in check

Checking repeatedly and stopping when the result looks significant inflates false positives. Committing to a sample size in advance removes the problem.

How to run it

  1. Compute required sample size before launch
  2. Write down the stopping rule
  3. Hide or ignore interim significance
  4. Read results only at the planned end
  5. Use sequential methods if early stopping is required

Ways to run it

Design options to test, not promised results.

  1. Option APre-registered sample size

    Compute the required sample before launch and write down the stopping rule.

  2. Option BHidden interim results

    Hide significance from the dashboard until the planned end so no one is tempted to stop early.

  3. Option CSequential testing

    If early stopping is a must, use a sequential method designed for repeated looks.

Timeline

  1. 12010 first documented: Evan Miller, 'How Not To Run An A/B Test'
  2. 22026-11 status checked: Working

Dated examples

No dated example yet. Treat this tactic as a hypothesis.

Does it still work?

Workingas of 2026-11

Classic 2010 essay; modern tools add sequential tests so peeking is safer (see sequential-testing). source

Risks

  • Business pressure to call winners early
  • Underpowered tests waste traffic

Sources

  1. evanmiller.org · evanmiller.org · Evan Miller · 2010-04-18
    “The more you peek, the more your significance levels will be off.”
    ✓ quote checked word-for-word on the page
  2. evanmiller.org · evanmiller.org · Evan Miller · 2010-04-18
    “Committing to a sample size completely mitigates the problem described here.”
    ✓ quote checked word-for-word on the page

Change log

  • 2026-11-20 — harvest v0.1 (ops-ai)