Working · 2026-11Vendor claimGrowth opsAnalytics

A/A tests to validate the setup

Run a test where both groups get the same experience to prove the experimentation pipeline works.

Also called: A/A test

Claim and evidence

The claimRun a test where both groups get the same experience to prove the experimentation pipeline works.

No dated example yet. Treat this tactic as a hypothesis.

What the numbers show

No published number for this tactic has passed our word-for-word check yet, so there is no chart.

Why it works

Diagram, not data
Both groups get thesame experienceOnly chance-leveldifferences: setup OKFrequent significantgaps: setup brokenFix and rerun
Both groups get the sameexperienceOnlychance-leveldifferences:setup OKFrequentsignificantgaps: setupbrokenFix and rerun

If an A/A test shows frequent significant differences, the SDK, logging or analysis is broken. A few chance hits across many metrics are expected.

How to run it

  1. Run an A/A after a new SDK integration or major change
  2. Confirm data reaches the warehouse
  3. Check that no metric moves significantly beyond chance
  4. Check for SRM and data issues
  5. Fix and rerun if the test fails

Ways to run it

Design options to test, not promised results.

  1. Option AAfter every new integration

    Run an A/A test after a new SDK integration or major pipeline change.

  2. Option BWarehouse check

    Use the A/A test to confirm experiment data actually reaches the warehouse.

  3. Option CSample ratio check

    Check that group sizes match the planned split before trusting any metric.

Dated examples

No dated example yet. Treat this tactic as a hypothesis.

Does it still work?

Workingas of 2026-11

Recommended in GrowthBook documentation; undated page. source

Risks

  • Costs traffic and calendar time
  • Passing an A/A does not prove later tests are valid

Sources

  1. docs.growthbook.io · GrowthBook docs · undated
    “They are used to test whether your experimentation set up and methodology is working as expected.”
    ✓ quote checked word-for-word on the page
  2. docs.growthbook.io · GrowthBook docs · undated
    “you can validate that the experiment data is being sent correctly to your data warehouse.”
    ✓ quote checked word-for-word on the page

Change log

  • 2026-11-20 — harvest v0.1 (ops-ai)