Claim and evidence
MAB is for optimization, not experimentation.Optimizely · support.optimizely.com · 2026-08-05
Vendor claim
A tool vendor reports numbers about its own customers or product. Useful, but not independent.
Working · 2026-10 Current Optimizely and Adobe documentation offer bandits as a standard option next to A/B tests.
What the numbers show
No published number for this tactic has passed our word-for-word check yet, so there is no chart.
Why it works
A fixed split sends a lot of traffic to losing variants until the test ends; a bandit cuts that cost by moving traffic to the leader as data comes in. The price is that you do not get a clean significance read, so bandits suit content that expires before an A/B test could finish, not lasting product decisions.
How to run it
- Use a bandit only when the goal is to maximise conversions during a short window
- Keep a classic A/B test for hypotheses you want to learn from
- Pick one primary metric for the algorithm to optimise
- Set a minimum exploration share so new variants still get traffic
- Do not report bandit winners as proven lifts
Ways to run it
Design options to test, not promised results.
- Option APromotions and headlines
Use a bandit for offers, headlines and calls to action that expire before a fixed test could finish, as Optimizely's docs suggest.
- Option BExploration floor
Keep a minimum share of traffic on every variant so new entries still get a fair look.
- Option CBandit then confirm
Let the bandit pick a leader for the campaign, then run a classic A/B test if the idea should become permanent.
Dated examples
- —
Optimizely
Optimizely's docs recommend bandits for promotions, headline tests and webinar sign-up calls to action, and say they do not produce statistical significance.
“MAB is for optimization, not experimentation.”
✓ quote checked word-for-word on the pagesupport.optimizely.com · 2026-08-05
Does it still work?
Current Optimizely and Adobe documentation offer bandits as a standard option next to A/B tests. source
Risks
- Easy to misuse for decisions that need statistical evidence
- Early noise can lock traffic onto a false winner
- Results are harder to explain to stakeholders
Sources
- support.optimizely.com
“You should never use MAB tests for exploratory hypotheses or variation selection.”
✓ quote checked word-for-word on the page - experienceleague.adobe.com
“A substantial proportion of traffic is directed toward inferior treatments, potentially reducing conversions or revenue during the test.”
✓ quote checked word-for-word on the page - support.optimizely.com
“It helps you capture more value from your experiments by reducing the time to statistical significance.”
✓ quote checked word-for-word on the page
Change log
- 2026-11-20 — harvest v0.1 (pass2-expansion-ops)