Claim and evidence
all members of our departments run and analyse more than a thousand concurrent experimentsBooking.com · ar5iv.arxiv.org · 2017
Primary
The company or founder describes the tactic and its result themselves: own blog, talk, filing.
Working · 2026-11 Paper is from 2017; a commonly repeated 25,000-tests-a-year figure was not verified here.
What the numbers show
run and analysed by staff across Booking.com departments (company paper, 2017)
ar5iv.arxiv.orgWhy it works
Broad ownership multiplies ideas tested. Guardrails and monitoring of business and system health keep that freedom from causing damage.
How to run it
- Build self-serve tooling
- Add automated health and impact checks
- Let any team member own an experiment end to end
- Give everyone the right to stop harmful tests
- Share results widely
Ways to run it
Design options to test, not promised results.
- Option ASelf-serve platform
Let any team member set up, run and analyse a test without a central gatekeeper, as at Booking.com.
- Option BAutomatic guardrails
Monitor business and system health metrics on every test and alert or stop when they degrade.
- Option COpen results feed
Publish every test and its outcome where the whole company can read and search them.
Timeline
- 12017 first documented: Kaufman, Pitchforth and Vermeer, Booking.com
- 22017 Booking.com — Staff across departments run and analyze experiments on a shared platform
- 32026-11 status checked: Working
Dated examples
- 2017
Booking.com
Staff across departments run and analyze experiments on a shared platform
“all members of our departments run and analyse more than a thousand concurrent experiments”
✓ quote checked word-for-word on the pagear5iv.arxiv.org · 2017
Does it still work?
Paper is from 2017; a commonly repeated 25,000-tests-a-year figure was not verified here. source
Risks
- Quality control dilutes without guardrails
- Needs heavy engineering investment
Sources
- ar5iv.arxiv.org
“Everyone in the company must be able and feel empowered to stop harmful experiments of their own accord.”
✓ quote checked word-for-word on the page - ar5iv.arxiv.org
“running experiments is cheap, safe and easy enough that anyone can go ahead with testing new ideas”
✓ quote checked word-for-word on the page
Change log
- 2026-11-20 — harvest v0.1 (ops-ai)