Claim and evidence
Estimate how long a test must run to detect the smallest effect you care about before launching it.
No dated example yet. Treat this tactic as a hypothesis.
Vendor claim
A tool vendor reports numbers about its own customers or product. Useful, but not independent.
Working · 2026-11 Built into platforms; the Statsig page notes that estimates beyond history are projections.
What the numbers show
No published number for this tactic has passed our word-for-word check yet, so there is no chart.
Why it works
Traffic and allocation determine the minimum detectable effect. Planning avoids running tests that could never detect a realistic lift.
How to run it
- Choose the metric and minimum effect worth detecting
- Use historical traffic to estimate exposures
- Compute duration and allocation
- Decide whether the test is feasible
- Pick bigger bets if traffic is too low
Ways to run it
Design options to test, not promised results.
- Option ACalculator before launch
Make a power calculation part of every test plan, using historical traffic for the chosen metric.
- Option BBigger bets on low traffic
When the detectable effect is unrealistically large, test bolder changes instead of small tweaks.
- Option CMore sensitive metric
Choose a metric closer to the change, or use variance reduction, so smaller effects become detectable.
Dated examples
No dated example yet. Treat this tactic as a hypothesis.
Does it still work?
Built into platforms; the Statsig page notes that estimates beyond history are projections. source
Risks
- Planning MDE is not a predicted effect
- Low-traffic products cannot test small changes
Sources
- docs.statsig.com
“The smallest change in the metric that the experiment can reliably detect.”
✓ quote checked word-for-word on the page - docs.statsig.com
“The target is a planning threshold, not a predicted effect.”
✓ quote checked word-for-word on the page
Change log
- 2026-11-20 — harvest v0.1 (ops-ai)