Working · 2026-10Vendor claimGrowth opsAnalytics

Multi-armed bandits for short-lived variants

For promotions, headlines and other short-lived content, let a bandit algorithm shift traffic to the best-performing variant during the test instead of holding a fixed A/B split.

Also called: MAB, bandit testing

Claim and evidence

The claimMAB is for optimization, not experimentation.

Optimizely · support.optimizely.com · 2026-08-05

What the numbers show

No published number for this tactic has passed our word-for-word check yet, so there is no chart.

Why it works

Diagram, not data
Is the contentshort-lived?Yes: bandit shiftstraffic to the leaderNo: classic A/B testto learnNever report banditwinners as provenlifts
Is the content short-lived?Yes: banditshifts trafficto the leaderNo: classic A/Btest to learnNever report bandit winners asproven lifts

A fixed split sends a lot of traffic to losing variants until the test ends; a bandit cuts that cost by moving traffic to the leader as data comes in. The price is that you do not get a clean significance read, so bandits suit content that expires before an A/B test could finish, not lasting product decisions.

How to run it

  1. Use a bandit only when the goal is to maximise conversions during a short window
  2. Keep a classic A/B test for hypotheses you want to learn from
  3. Pick one primary metric for the algorithm to optimise
  4. Set a minimum exploration share so new variants still get traffic
  5. Do not report bandit winners as proven lifts

Ways to run it

Design options to test, not promised results.

  1. Option APromotions and headlines

    Use a bandit for offers, headlines and calls to action that expire before a fixed test could finish, as Optimizely's docs suggest.

  2. Option BExploration floor

    Keep a minimum share of traffic on every variant so new entries still get a fair look.

  3. Option CBandit then confirm

    Let the bandit pick a leader for the campaign, then run a classic A/B test if the idea should become permanent.

Dated examples

  1. —

    Optimizely

    Optimizely's docs recommend bandits for promotions, headline tests and webinar sign-up calls to action, and say they do not produce statistical significance.

    “MAB is for optimization, not experimentation.”
    ✓ quote checked word-for-word on the page

    support.optimizely.com · 2026-08-05

Does it still work?

Workingas of 2026-10

Current Optimizely and Adobe documentation offer bandits as a standard option next to A/B tests. source

Risks

  • Easy to misuse for decisions that need statistical evidence
  • Early noise can lock traffic onto a false winner
  • Results are harder to explain to stakeholders

Sources

  1. support.optimizely.com · Optimizely Feature Experimentation docs · 2026-08-05
    “You should never use MAB tests for exploratory hypotheses or variation selection.”
    ✓ quote checked word-for-word on the page
  2. experienceleague.adobe.com · Adobe Experience League · 2026-10-06
    “A substantial proportion of traffic is directed toward inferior treatments, potentially reducing conversions or revenue during the test.”
    ✓ quote checked word-for-word on the page
  3. support.optimizely.com · Optimizely Experimentation Strategy docs · 2026-08-05
    “It helps you capture more value from your experiments by reducing the time to statistical significance.”
    ✓ quote checked word-for-word on the page

Change log

  • 2026-11-20 — harvest v0.1 (pass2-expansion-ops)