Glossaryrobots.txt token
What is robots.txt token?
A robots.txt token is the name an operator tells you to use in the User-agent line of robots.txt to address one of its crawlers. It usually matches the name in the crawler’s user agent string, but not always, and some tokens exist only in robots.txt.
Also called: robots.txt user-agent name, robots.txt name.
Rules in robots.txt are per token. A Disallow under GPTBot says nothing to OAI-SearchBot or ChatGPT-User, even though all three are OpenAI. That is how sites end up blocking ChatGPT citations while trying to opt out of training, or the reverse.
Some tokens never appear as visitors. Google-Extended controls whether Google uses your pages for Gemini training, but the crawling is done by plain Googlebot; Applebot-Extended works the same way for Apple. You set the rule, and the operator applies it on their side.
robots.txt is a request, not a wall. It is honoured by crawlers that choose to honour it and enforced by nobody. A crawler that ignores it, or a scraper using a spoofed user agent, is only stopped by server rules.
Related terms
- User agent string: A user agent string is the text a client sends with every web request to say what it is.
- Training crawler: A training crawler collects web pages to use as training data for AI models.
- AI search crawler: An AI search crawler is a crawler that builds the index an AI assistant searches and cites from.
Published 2026-09-21, updated 2026-09-21.