Claim and evidence
Monitor metrics you must not break (errors, speed, retention) alongside the primary metric of every experiment.
No dated example yet. Treat this tactic as a hypothesis.
Vendor claim
A tool vendor reports numbers about its own customers or product. Useful, but not independent.
Working · 2026-11 Standard in platforms from Spotify Confidence and GrowthBook.
What the numbers show
No published number for this tactic has passed our word-for-word check yet, so there is no chart.
Why it works
A test can win its target metric while hurting something else. Guardrails catch regressions before the change ships.
How to run it
- Pick a few critical guardrail metrics
- Set acceptable thresholds
- Monitor them during the test
- Stop or roll back on regression
- Review guardrails per product area
Ways to run it
Design options to test, not promised results.
- Option ATechnical guardrails
Track error rate and page load time on every test alongside the primary metric.
- Option BRetention guardrail
Add a retention or cancellation metric so short-term wins do not hide longer-term harm.
- Option CThresholds with auto-alert
Set an acceptable threshold for each guardrail and alert the owner when a test crosses it.
Dated examples
No dated example yet. Treat this tactic as a hypothesis.
Does it still work?
Standard in platforms from Spotify Confidence and GrowthBook. source
Risks
- Too many guardrails cause false alarms
- Weak power on guardrails can miss a real regression
Sources
- confidence.spotify.com
“Guardrail metrics define what you're trying not to break.”
✓ quote checked word-for-word on the page - docs.growthbook.io
“Guardrail Metrics help ensure an experiment isn't inadvertently hurting core metrics like error rate or page load time”
✓ quote checked word-for-word on the page
Change log
- 2026-11-20 — harvest v0.1 (ops-ai)