Glossary · 18 terms
AI traffic, in plain language.
AI agents are a new kind of visitor, and they came with a new vocabulary. If you work in analytics or SEO, these are the words you will keep running into, explained in plain language.
Definitions
A to Z
- AI agent
- An AI agent is any automated visitor that fetches web pages on behalf of an AI product: the crawler that collects training data, the crawler that builds an AI search index, and the fetcher that reads a page live while a person is talking to an assistant.
- AI citationalso: AI referral, assistant citation
- An AI citation is an assistant naming or linking your page in its answer. It is the outcome that AI search crawling and live fetching lead to, and the only part of the chain that can send a person to your site.
- AI crawleralso: AI bot, LLM crawler
- An AI crawler is an automated program that visits pages on its own schedule to collect them for an AI product, either as training data or to build a search index. Unlike a live fetch, a crawl is not triggered by a person asking a question.
- AI search crawleralso: AI search index crawler
- An AI search crawler is a crawler that builds the index an AI assistant searches and cites from. It visits ahead of any question, so its visits do not correspond to readers, but being in its index is what makes you citable.
- Bot verificationalso: crawler verification, verified bot
- Bot verification is checking that a request claiming to be a known bot really came from its operator, by comparing the request’s IP address with the addresses the operator publishes, or by reverse DNS when no list exists.
- Dark social
- Dark social is sharing that analytics cannot attribute: links passed in private messages, chat apps, email and text, which arrive with no referrer and get filed under "direct" traffic.
- Flooralso: floor count, minimum count
- A floor is a count reported as "at least this many": the number that could be measured, with the acknowledgement that the true number may be higher. agenttraffic labels a count as a floor whenever it could be undercounted by hiding, spoofing or caching.
- Generative engine optimization (GEO)also: GEO, AI SEO, answer engine optimization, AEO
- Generative engine optimization is the practice of making a site more likely to be found, read and cited by AI assistants, the way search engine optimization does for search engines.
- Link unfurleralso: unfurler, link preview fetcher, unfurl
- A link unfurler is the fetcher a chat app or social platform sends to build a preview when someone pastes a link. Slack, Discord, iMessage, WhatsApp and Facebook all have one. Each unfurl is evidence that a person shared your URL somewhere analytics cannot see.
- Live fetchalso: live fetch agent, user-triggered fetch, on-demand fetch
- A live fetch is a request an AI assistant makes for a page right now, because a person in a conversation asked something the assistant needs that page to answer. Each live fetch stands for one reader, arriving through an assistant instead of a browser.
- Published IP rangesalso: crawler IP list, bot IP addresses
- Published IP ranges are the lists of IP addresses an operator makes public for its crawlers and agents, so site owners can confirm that a request wearing the operator’s name really came from them.
- Reverse DNS verificationalso: rDNS verification
- Reverse DNS verification confirms a crawler by looking up the hostname for its IP address, checking that the hostname belongs to the operator’s domain, and then resolving that hostname forward to make sure it points back to the same IP.
- robots.txt tokenalso: robots.txt user-agent name, robots.txt name
- A robots.txt token is the name an operator tells you to use in the User-agent line of robots.txt to address one of its crawlers. It usually matches the name in the crawler’s user agent string, but not always, and some tokens exist only in robots.txt.
- Search engine crawler
- A search engine crawler indexes pages for a traditional web search engine. Googlebot and bingbot are the main ones, and their indexes now also feed the AI answers shown on results pages and in assistants that use them.
- Server logalso: access log, CDN log, edge log
- A server log (or access log) is the file a web server or CDN writes with one line per request it answered: time, path, status, client IP, user agent and referrer. It is the only record that includes every visitor, human or not, whether or not they ran any script.
- Spoofed user agentalso: spoofed bot, fake bot, bot impersonation
- A spoofed user agent is a request that claims to be a known bot, such as Googlebot or ChatGPT-User, but comes from an IP address outside the ranges that bot’s operator publishes. It is a scraper or scanner borrowing a trusted name.
- Training crawler
- A training crawler collects web pages to use as training data for AI models. Its visits do not correspond to readers or citations, and blocking it costs no visibility in search or assistants.
- User agent stringalso: user agent, UA string, UA
- A user agent string is the text a client sends with every web request to say what it is. Browsers, crawlers and assistants all send one, and it is how a log line reveals which AI agent made a request.
See it in your logs
One command, nothing to install
Point it at CloudFront, nginx or Apache logs to see which of these visitors are already on your site.
$ npx agenttraffic ./logs --days 7Quick start
Roadmap
Get a weekly report in your inbox
The command-line tool is free and stays free. A hosted version that emails you each week is on the roadmap. Leave your email and you'll hear once, when it exists.