GlossaryAI search crawler
What is AI search crawler?
An AI search crawler is a crawler that builds the index an AI assistant searches and cites from. It visits ahead of any question, so its visits do not correspond to readers, but being in its index is what makes you citable.
Also called: AI search index crawler.
OAI-SearchBot builds the index behind ChatGPT search, PerplexityBot the one behind Perplexity, Claude-SearchBot the one behind Claude. Each has its own name and its own robots.txt rule, separate from the operator’s training crawler.
This is the crawler to allow if you want to appear in AI answers. Blocking it removes your pages from that assistant’s search results and citations, while doing nothing about training. Sites that block "all AI bots" to opt out of training often block this one by accident.
In the report, AI search crawler visits are listed next to live fetch visits under AI crawlers, because together they describe how an assistant finds and then reads your pages.
AI search agents in the registry
- OAI-SearchBot (OpenAI)
Related terms
- AI crawler: An AI crawler is an automated program that visits pages on its own schedule to collect them for an AI product, either as training data or to build a search index.
- Training crawler: A training crawler collects web pages to use as training data for AI models.
- Live fetch: A live fetch is a request an AI assistant makes for a page right now, because a person in a conversation asked something the assistant needs that page to answer.
- AI citation: An AI citation is an assistant naming or linking your page in its answer.
Published 2026-09-21, updated 2026-09-21.