Guides
Telling the agents apart.
Every operator runs several agents under similar names, and a robots.txt rule only applies to the one name it mentions. These guides put the confusable ones side by side so you block the right one.
Comparisons
One operator, three agents, three consequences
By kind
Start from what a visit means
Names change and operators add agents, but there are only a few jobs an agent can be doing. Each kind has its own page listing every agent that does that job.
Words
The five terms behind most wrong robots.txt files
- Live fetch
- A live fetch is a request an AI assistant makes for a page right now, because a person in a conversation asked something the assistant needs that page to answer.
- AI search crawler
- An AI search crawler is a crawler that builds the index an AI assistant searches and cites from.
- Training crawler
- A training crawler collects web pages to use as training data for AI models.
- robots.txt token
- A robots.txt token is the name an operator tells you to use in the User-agent line of robots.txt to address one of its crawlers.
- Spoofed user agent
- A spoofed user agent is a request that claims to be a known bot, such as Googlebot or ChatGPT-User, but comes from an IP address outside the ranges that bot’s operator publishes.
The whole glossary, 18 terms, each on its own page.
See it in your logs
One command, nothing to install
Point it at CloudFront, nginx or Apache logs. The report lists each of these agents on its own row, so you can see which one actually visits.
Roadmap
Get a weekly report in your inbox
The command-line tool is free and stays free. A hosted version that emails you each week is on the roadmap. Leave your email and you'll hear once, when it exists.