GlossaryTraining crawler

What is training crawler?

A training crawler collects web pages to use as training data for AI models. Its visits do not correspond to readers or citations, and blocking it costs no visibility in search or assistants.

GPTBot, ClaudeBot, Bytespider, Amazonbot and CCBot are training crawlers. They visit on their own schedule, often at volume, and the pages they collect may appear in a model’s knowledge months later without any link back.

Most operators publish a robots.txt name for their training crawler, and some publish an opt-out name that never appears as a visitor at all: Google-Extended and Applebot-Extended are robots.txt rules, not user agents, so you will never see them in a log. Gemini training traffic arrives as plain Googlebot.

agenttraffic hides training crawlers from the page lists by default so they do not drown out the visits that represent people. The --show-training flag brings them back.

Training agents in the registry

All training agents.

Related terms

  • AI crawler: An AI crawler is an automated program that visits pages on its own schedule to collect them for an AI product, either as training data or to build a search index.
  • AI search crawler: An AI search crawler is a crawler that builds the index an AI assistant searches and cites from.
  • robots.txt token: A robots.txt token is the name an operator tells you to use in the User-agent line of robots.txt to address one of its crawlers.

Published 2026-09-21, updated 2026-09-21.

See it in your logs

One command, nothing to install

Point it at CloudFront, nginx or Apache logs. Every AI agent that visited gets its own row, verified against the addresses its operator publishes.

$ npx agenttraffic ./logs --days 7Quick start

Roadmap

Get a weekly report in your inbox

The command-line tool is free and stays free. A hosted version that emails you each week is on the roadmap. Leave your email and you'll hear once, when it exists.