Working · 2026-11PrimaryGrowth opsProcess

Democratized experimentation with safeguards

Make experiments cheap and safe enough that anyone can run them, with automated safeguards and the right to stop harmful tests.

Also called: experiment-for-everyone

Claim and evidence

The claimall members of our departments run and analyse more than a thousand concurrent experiments

Booking.com · ar5iv.arxiv.org · 2017

What the numbers show

more than a thousand

run and analysed by staff across Booking.com departments (company paper, 2017)

ar5iv.arxiv.org

Why it works

Diagram, not data
Anyone launches atestAutomated healthchecks watch itAnyone can stopharmful testsResults sharedwidely
Anyonelaunches atestAutomatedhealth checkswatch itAnyone canstop harmfultestsResultsshared widely

Broad ownership multiplies ideas tested. Guardrails and monitoring of business and system health keep that freedom from causing damage.

How to run it

  1. Build self-serve tooling
  2. Add automated health and impact checks
  3. Let any team member own an experiment end to end
  4. Give everyone the right to stop harmful tests
  5. Share results widely

Ways to run it

Design options to test, not promised results.

  1. Option ASelf-serve platform

    Let any team member set up, run and analyse a test without a central gatekeeper, as at Booking.com.

  2. Option BAutomatic guardrails

    Monitor business and system health metrics on every test and alert or stop when they degrade.

  3. Option COpen results feed

    Publish every test and its outcome where the whole company can read and search them.

Timeline

  1. 12017 first documented: Kaufman, Pitchforth and Vermeer, Booking.com
  2. 22017 Booking.com — Staff across departments run and analyze experiments on a shared platform
  3. 32026-11 status checked: Working

Dated examples

  1. 2017

    Booking.com

    Staff across departments run and analyze experiments on a shared platform

    “all members of our departments run and analyse more than a thousand concurrent experiments”
    ✓ quote checked word-for-word on the page

    ar5iv.arxiv.org · 2017

Does it still work?

Workingas of 2026-11

Paper is from 2017; a commonly repeated 25,000-tests-a-year figure was not verified here. source

Risks

  • Quality control dilutes without guardrails
  • Needs heavy engineering investment

Sources

  1. ar5iv.arxiv.org · arXiv (Booking.com) · Raphael Lopez Kaufman, Jegar Pitchforth, Lukas Vermeer · 2017
    “Everyone in the company must be able and feel empowered to stop harmful experiments of their own accord.”
    ✓ quote checked word-for-word on the page
  2. ar5iv.arxiv.org · arXiv (Booking.com) · Raphael Lopez Kaufman, Jegar Pitchforth, Lukas Vermeer · 2017
    “running experiments is cheap, safe and easy enough that anyone can go ahead with testing new ideas”
    ✓ quote checked word-for-word on the page

Change log

  • 2026-11-20 — harvest v0.1 (ops-ai)