Claim and evidence
As we suspected, the personalization worked and generated a significant lift in our core metrics.Netflix · reforge.com · 2020-10-02
Secondary
A reputable third party analysed it (Lenny's Newsletter, First Round Review, Reforge, an Ahrefs study).
Netflix reported a significant lift in core metrics; no figure given in the summary.
Working · 2026-08 Optimizely's current docs offer contextual bandits for pages whose content changes rarely; the Netflix case is older and was read through a Reforge summary because the Netflix blog blocked direct fetching.
What the numbers show
No published number for this tactic has passed our word-for-word check yet, so there is no chart.
Why it works
Different users respond to different versions of the same content, so one global winner leaves value on the table. A contextual bandit keeps a little controlled randomness to keep learning while serving each context its likely best variant, and quality-of-engagement metrics stop it from learning clickbait.
How to run it
- Prepare a varied pool of creative for each item
- Choose context features (user history, device, page)
- Reward deeper engagement, not just clicks
- Evaluate candidate models offline on logged data before going live
- Confirm the lift with an A/B test against an unpersonalized bandit
Before / after
Generalised wireframe of the pattern, not a copy of any company's screen.
Ways to run it
Design options to test, not promised results.
- Option APer-user artwork
Choose which image of the same item each member sees based on their history, as Netflix does with title artwork.
- Option BOffline replay first
Evaluate candidate models on logged data before going live, then confirm with an A/B test.
- Option CEngagement-quality reward
Reward deeper engagement rather than clicks so the model does not learn clickbait.
- Option DStable pages only
Use it on pages whose content does not change often, such as the home page.
Dated examples
- —
Netflix
Netflix moved from finding one best artwork per title with bandits to choosing artwork per member with contextual bandits, evaluated offline by replay and then A/B tested against unpersonalized bandits.
Netflix reported a significant lift in core metrics; no figure given in the summary.
“As we suspected, the personalization worked and generated a significant lift in our core metrics.”
✓ quote checked word-for-word on the pagereforge.com · 2020-10-02
Does it still work?
Optimizely's current docs offer contextual bandits for pages whose content changes rarely; the Netflix case is older and was read through a Reforge summary because the Netflix blog blocked direct fetching. source
Risks
- Needs large traffic and engineering for low-latency serving
- Can learn clickbait if rewarded on clicks
- Changing creative a user already knows can hurt recognisability
Sources
- reforge.com
“We also look at the quality of engagement to avoid learning a model that recommends “clickbait” images”
✓ quote checked word-for-word on the page - support.optimizely.com
“Contextual bandits are ideal to be run on pages where the content is not changed too frequently, such as the homepage”
✓ quote checked word-for-word on the page
Change log
- 2026-11-20 — harvest v0.1 (pass2-expansion-ops)