Working · 2026-11PrimaryAcquisitionAEO / AI search

Allow AI search crawlers

Make sure robots.txt allows search-specific AI crawlers such as OAI-SearchBot so the site can appear in ChatGPT search answers.

Also called: Allow AI search crawlers

Claim and evidence

The claimMake sure robots.txt allows search-specific AI crawlers such as OAI-SearchBot so the site can appear in ChatGPT search answers.

No dated example yet. Treat this tactic as a hypothesis.

What the numbers show

5.89%

of all websites block GPTBot, the most blocked AI bot (Ahrefs data)

ahrefs.com

Why it works

Diagram, not data
Review robots.txtAllow AI search botsBlock AI search botsTraining bots: aseparate call
Review robots.txtAllow AI searchbotsBlock AI searchbotsTraining bots: a separate call

OpenAI states sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers; GPTBot (training) is separate, so you can allow search and block training.

How to run it

  1. Review robots.txt for OAI-SearchBot and other AI search bots
  2. Decide separately on training bots like GPTBot
  3. Allow search bots and wait about 24 hours
  4. Verify in server logs

Ways to run it

Design options to test, not promised results.

  1. Option ASearch yes, training no

    Allow search crawlers such as OAI-SearchBot while keeping training crawlers such as GPTBot blocked.

  2. Option BAllow all AI bots

    Open the site to both search and training crawlers when broad exposure matters more than control.

  3. Option CLog-verified rollout

    Change robots.txt, then confirm in server logs that the search bots actually reach key pages.

Dated examples

No dated example yet. Treat this tactic as a hypothesis.

Does it still work?

Workingas of 2026-11

OpenAI bots documentation source

Risks

  • Allowing is necessary, not sufficient for citation
  • Policy per vendor changes

Sources

  1. developers.openai.com · OpenAI Developers · undated
    “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.”
    ✓ quote checked word-for-word on the page
  2. ahrefs.com · Ahrefs blog · Patrick Stox · 2025-05-21
    “GPTBot (OpenAI) is the most blocked AI bot, with 5.89% of all websites blocking them.”
    ✓ quote checked word-for-word on the page

Change log

  • 2026-11-20 — merged: pass2-enrich.jsonl:12 (allow-ai-search-crawlers); acq-search.jsonl:35 (allow-ai-search-crawlers)
  • 2026-11-20 — harvest v0.1 (pass2-enrich, acq-search)