Working · 2026-11Vendor claimGrowth opsAnalytics

Power analysis and MDE planning

Estimate how long a test must run to detect the smallest effect you care about before launching it.

Also called: sample size calculation

Claim and evidence

The claimEstimate how long a test must run to detect the smallest effect you care about before launching it.

No dated example yet. Treat this tactic as a hypothesis.

What the numbers show

No published number for this tactic has passed our word-for-word check yet, so there is no chart.

Why it works

Diagram, not data
Pick themetric andsmallesteffectEstimatetraffic anddurationFeasible:launch thetestToo slow: picka bigger betNo test thatcould neverdetect a lift
Pick the metric and smallesteffectEstimate traffic and durationFeasible:launch the testToo slow: picka bigger betNo test that could never detecta lift

Traffic and allocation determine the minimum detectable effect. Planning avoids running tests that could never detect a realistic lift.

How to run it

  1. Choose the metric and minimum effect worth detecting
  2. Use historical traffic to estimate exposures
  3. Compute duration and allocation
  4. Decide whether the test is feasible
  5. Pick bigger bets if traffic is too low

Ways to run it

Design options to test, not promised results.

  1. Option ACalculator before launch

    Make a power calculation part of every test plan, using historical traffic for the chosen metric.

  2. Option BBigger bets on low traffic

    When the detectable effect is unrealistically large, test bolder changes instead of small tweaks.

  3. Option CMore sensitive metric

    Choose a metric closer to the change, or use variance reduction, so smaller effects become detectable.

Dated examples

No dated example yet. Treat this tactic as a hypothesis.

Does it still work?

Workingas of 2026-11

Built into platforms; the Statsig page notes that estimates beyond history are projections. source

Risks

  • Planning MDE is not a predicted effect
  • Low-traffic products cannot test small changes

Sources

  1. docs.statsig.com · Statsig docs · undated
    “The smallest change in the metric that the experiment can reliably detect.”
    ✓ quote checked word-for-word on the page
  2. docs.statsig.com · Statsig docs · undated
    “The target is a planning threshold, not a predicted effect.”
    ✓ quote checked word-for-word on the page

Change log

  • 2026-11-20 — harvest v0.1 (ops-ai)