Working · 2026-11PrimaryGrowth opsAnalytics

Check for novelty and primacy effects

Account for effects that fade (novelty) or grow (primacy) over time before treating early lifts as permanent.

Also called: novelty effect

Claim and evidence

The claimAccount for effects that fade (novelty) or grow (primacy) over time before treating early lifts as permanent.

No dated example yet. Treat this tactic as a hypothesis.

What the numbers show

No published number for this tactic has passed our word-for-word check yet, so there is no chart.

Why it works

Diagram, not data
Plot the treatmenteffect by dayEffect still moving:extend the testEffect stable: readthe resultConfirm the winnerwith a holdout
Plot the treatment effect by dayEffect stillmoving: extendthe testEffect stable:read the resultConfirm the winner with aholdout

Users react to something new, then settle, or learn a change and use it more later. Reading a test too early captures the temporary effect.

How to run it

  1. Plot the treatment effect by day
  2. Compare new versus returning users
  3. Extend the test if the effect is still moving
  4. Confirm winners with a holdout
  5. Prefer long-term estimators for big UI changes

Ways to run it

Design options to test, not promised results.

  1. Option ANew versus returning users

    Compare the effect for first-time users with returning ones; novelty shows up mostly in the returning group.

  2. Option BLonger run for big UI changes

    Run large interface changes longer than usual so learning and fading effects play out.

  3. Option CPost-launch holdout

    Keep a small group on the old version after launch to check whether the lift persists.

Timeline

  1. 12021 first documented: Sadeghi et al., Microsoft researchers, arXiv
  2. 22026-11 status checked: Working

Dated examples

No dated example yet. Treat this tactic as a hypothesis.

Does it still work?

Workingas of 2026-11

Described in a 2021 Microsoft paper; practical guidance such as two-week minimums comes from practitioner blogs, not fetched. source

Risks

  • Hard to tell a novelty effect from noise in short tests
  • No universal duration rule

Sources

  1. arxiv.org · arXiv · Soheil Sadeghi, Somit Gupta, Stefan Gramatovici, Jiannan Lu, Hao Ai, Ruhan Zhang · 2021-02-18
    “Novelty describes the desire to use new technology that tends to diminish over time.”
    ✓ quote checked word-for-word on the page
  2. arxiv.org · arXiv · Soheil Sadeghi, Somit Gupta, Stefan Gramatovici, Jiannan Lu, Hao Ai, Ruhan Zhang · 2021-02-18
    “Primacy describes the growing engagement with technology as a result of adoption of the innovation.”
    ✓ quote checked word-for-word on the page

Change log

  • 2026-11-20 — harvest v0.1 (ops-ai)