Claim and evidence
Run a test where both groups get the same experience to prove the experimentation pipeline works.
No dated example yet. Treat this tactic as a hypothesis.
Vendor claim
A tool vendor reports numbers about its own customers or product. Useful, but not independent.
Working · 2026-11 Recommended in GrowthBook documentation; undated page.
What the numbers show
No published number for this tactic has passed our word-for-word check yet, so there is no chart.
Why it works
If an A/A test shows frequent significant differences, the SDK, logging or analysis is broken. A few chance hits across many metrics are expected.
How to run it
- Run an A/A after a new SDK integration or major change
- Confirm data reaches the warehouse
- Check that no metric moves significantly beyond chance
- Check for SRM and data issues
- Fix and rerun if the test fails
Ways to run it
Design options to test, not promised results.
- Option AAfter every new integration
Run an A/A test after a new SDK integration or major pipeline change.
- Option BWarehouse check
Use the A/A test to confirm experiment data actually reaches the warehouse.
- Option CSample ratio check
Check that group sizes match the planned split before trusting any metric.
Dated examples
No dated example yet. Treat this tactic as a hypothesis.
Does it still work?
Recommended in GrowthBook documentation; undated page. source
Risks
- Costs traffic and calendar time
- Passing an A/A does not prove later tests are valid
Sources
- docs.growthbook.io
“They are used to test whether your experimentation set up and methodology is working as expected.”
✓ quote checked word-for-word on the page - docs.growthbook.io
“you can validate that the experiment data is being sent correctly to your data warehouse.”
✓ quote checked word-for-word on the page
Change log
- 2026-11-20 — harvest v0.1 (ops-ai)