Claim and evidence
Make sure robots.txt allows search-specific AI crawlers such as OAI-SearchBot so the site can appear in ChatGPT search answers.
No dated example yet. Treat this tactic as a hypothesis.
Primary
The company or founder describes the tactic and its result themselves: own blog, talk, filing.
Working · 2026-11 OpenAI bots documentation
What the numbers show
of all websites block GPTBot, the most blocked AI bot (Ahrefs data)
ahrefs.comWhy it works
OpenAI states sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers; GPTBot (training) is separate, so you can allow search and block training.
How to run it
- Review robots.txt for OAI-SearchBot and other AI search bots
- Decide separately on training bots like GPTBot
- Allow search bots and wait about 24 hours
- Verify in server logs
Ways to run it
Design options to test, not promised results.
- Option ASearch yes, training no
Allow search crawlers such as OAI-SearchBot while keeping training crawlers such as GPTBot blocked.
- Option BAllow all AI bots
Open the site to both search and training crawlers when broad exposure matters more than control.
- Option CLog-verified rollout
Change robots.txt, then confirm in server logs that the search bots actually reach key pages.
Dated examples
No dated example yet. Treat this tactic as a hypothesis.
Does it still work?
OpenAI bots documentation source
Risks
- Allowing is necessary, not sufficient for citation
- Policy per vendor changes
Sources
- developers.openai.com
“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.”
✓ quote checked word-for-word on the page - ahrefs.com
“GPTBot (OpenAI) is the most blocked AI bot, with 5.89% of all websites blocking them.”
✓ quote checked word-for-word on the page
Change log
- 2026-11-20 — merged: pass2-enrich.jsonl:12 (allow-ai-search-crawlers); acq-search.jsonl:35 (allow-ai-search-crawlers)
- 2026-11-20 — harvest v0.1 (pass2-enrich, acq-search)