GlossaryAI agent
What is AI agent?
An AI agent is any automated visitor that fetches web pages on behalf of an AI product: the crawler that collects training data, the crawler that builds an AI search index, and the fetcher that reads a page live while a person is talking to an assistant.
The word covers several very different visitors. What they have in common is that none of them is a person in a browser, so none of them runs the tracking script that web analytics depends on. They leave their only trace in the server log.
agenttraffic sorts AI agents by what a visit means rather than by who runs them: a live fetch means a person just asked about the page, an ai search crawler means the page is being made findable in AI search, a training crawler means the page was collected as training data. The same operator usually runs one of each, under different names.
Search engine crawlers like Googlebot and bingbot are listed alongside AI agents because their index now feeds AI answers too, and link unfurlers are listed because their visits are evidence of people sharing links.
Related terms
- AI crawler: An AI crawler is an automated program that visits pages on its own schedule to collect them for an AI product, either as training data or to build a search index.
- Live fetch: A live fetch is a request an AI assistant makes for a page right now, because a person in a conversation asked something the assistant needs that page to answer.
- AI search crawler: An AI search crawler is a crawler that builds the index an AI assistant searches and cites from.
- Training crawler: A training crawler collects web pages to use as training data for AI models.
- User agent string: A user agent string is the text a client sends with every web request to say what it is.
Published 2026-09-21, updated 2026-09-21.