Working · 2018-11PrimaryRetentionAnalytics

'Very disappointed' survey as retention engine

Survey users on how they would feel without the product, segment the happy ones, and split the roadmap between strengths and blockers.

Also called: Sean Ellis test, PMF engine

Claim and evidence

The claimspend half your time doubling down on what users already love and the other half on addressing what’s holding others back.

Superhuman · review.firstround.com · 2018-11-12

What the numbers show

Share of Superhuman users who would be very disappointed without the product

22%58%

Why it works

Diagram, not data
Ask how they'dfeel withoutitProfile theverydisappointedDouble down onwhat they loveFix what holdsothers backRe-survey eachquarter
Ask how they'd feel without itProfile the very disappointedDouble down onwhat they loveFix what holdsothers backRe-survey each quarter

It identifies the real supporter segment and the specific objections of users who would be only somewhat disappointed, giving a prioritized retention roadmap.

How to run it

  1. Ask how they'd feel if they could no longer use the product
  2. Filter to 'very disappointed' to profile supporters
  3. Analyze objections of the others
  4. Split roadmap half-and-half and re-survey each quarter

Before / after

Before
How satisfied are you with Product?
Anything else?
Send
Generic satisfaction question
After
How would you feel if you could no longer use Product?
Very disappointed
Somewhat disappointed
Not disappointed
What would you use instead?
Send
One dependence question plus why

Generalised wireframe of the pattern, not a copy of any company's screen.

Ways to run it

Design options to test, not promised results.

  1. Option AHalf-and-half roadmap

    Spend half the roadmap on what supporters love and half on objections of the somewhat disappointed, as Superhuman did.

  2. Option BSupporter profile

    Filter to the very disappointed and describe who they are, then aim acquisition at people like them.

  3. Option CQuarterly re-survey

    Run the same question every quarter and track the share of very disappointed users over time.

Dated examples

  1. 2018

    Superhuman

    Founder ran the survey, segmented respondents and steered the roadmap.

    Moved the score from 22% to 58% within three quarters (per summary).

    “spend half your time doubling down on what users already love and the other half on addressing what’s holding others back.”
    ✓ quote checked word-for-word on the page

    review.firstround.com · 2018-11-12

Does it still work?

Workingas of 2018-11

Widely replicated method; source is 2018. source

Risks

  • Small samples
  • Survey measures sentiment, not behavior

Sources

  1. review.firstround.com · First Round Review · Rahul Vohra · 2018-11-12
    “spend half your time doubling down on what users already love and the other half on addressing what’s holding others back.”
    ✓ quote checked word-for-word on the page

Change log

  • 2026-11-20 — harvest v0.1 (retention)