GlossaryAI crawler
What is AI crawler?
An AI crawler is an automated program that visits pages on its own schedule to collect them for an AI product, either as training data or to build a search index. Unlike a live fetch, a crawl is not triggered by a person asking a question.
Also called: AI bot, LLM crawler.
Crawlers announce themselves in the user agent string (GPTBot, ClaudeBot, PerplexityBot, Bytespider) and most publish a robots.txt name you can allow or block. Their visits arrive in bursts, cover many pages, and do not correspond to readers.
The two crawler kinds are worth telling apart. A training crawler collects pages for model training, and blocking it costs you nothing visible. An ai search crawler builds the index an assistant cites from, and blocking it removes you from those answers. Operators usually run both under different names, which is why one robots.txt rule rarely does what people expect.
Some crawling is undeclared. Perplexity, for example, has been documented fetching pages under generic browser user agents. That is why counts of any crawler are a floor.
Related terms
- AI agent: An AI agent is any automated visitor that fetches web pages on behalf of an AI product: the crawler that collects training data, the crawler that builds an AI search index, and the fetcher that reads a page live while a person is talking to an assistant..
- Training crawler: A training crawler collects web pages to use as training data for AI models.
- AI search crawler: An AI search crawler is a crawler that builds the index an AI assistant searches and cites from.
- robots.txt token: A robots.txt token is the name an operator tells you to use in the User-agent line of robots.txt to address one of its crawlers.
- Spoofed user agent: A spoofed user agent is a request that claims to be a known bot, such as Googlebot or ChatGPT-User, but comes from an IP address outside the ranges that bot’s operator publishes.
Published 2026-09-21, updated 2026-09-21.