GlossarySearch engine crawler

What is search engine crawler?

A search engine crawler indexes pages for a traditional web search engine. Googlebot and bingbot are the main ones, and their indexes now also feed the AI answers shown on results pages and in assistants that use them.

These are the oldest bots on the web and the most impersonated. Both Google and Microsoft publish the IP addresses their crawlers use, so a request claiming to be Googlebot can be checked. In a typical week a site sees a handful of spoofed user agent requests wearing the Googlebot name.

Bing matters more than it used to. Several assistants, including ChatGPT and Copilot, draw on Bing’s index, so bingbot visits are part of how you become citable in AI answers.

Related terms

  • AI search crawler: An AI search crawler is a crawler that builds the index an AI assistant searches and cites from.
  • Spoofed user agent: A spoofed user agent is a request that claims to be a known bot, such as Googlebot or ChatGPT-User, but comes from an IP address outside the ranges that bot’s operator publishes.
  • Published IP ranges: Published IP ranges are the lists of IP addresses an operator makes public for its crawlers and agents, so site owners can confirm that a request wearing the operator’s name really came from them..

Published 2026-09-21, updated 2026-09-21.

See it in your logs

One command, nothing to install

Point it at CloudFront, nginx or Apache logs. Every AI agent that visited gets its own row, verified against the addresses its operator publishes.

$ npx agenttraffic ./logs --days 7Quick start

Roadmap

Get a weekly report in your inbox

The command-line tool is free and stays free. A hosted version that emails you each week is on the roadmap. Leave your email and you'll hear once, when it exists.