What are AI crawlers?

How AI crawlers work

AI companies run separate bots for separate jobs. A training bot collects pages that may be used to train future models. A search bot reads pages so the company's AI search can find and link them. A user bot fetches one page when a person asks the assistant about it.

Each bot announces itself with a name, called a user agent. A site's robots.txt file, a plain-text file at the root of the domain, lists rules by user agent, so a business can let the search bots in and keep the training bots out.

# robots.txt for example.com

# Let AI search crawlers read the site
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Opt out of AI model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Where each company documents its crawlers

In OpenAI's crawler documentation, GPTBot collects content that may train its models, OAI-SearchBot surfaces websites in ChatGPT's search results, and ChatGPT-User visits a page when a person's request calls for it, where OpenAI says robots.txt rules may not apply.

Anthropic's help article lists three: ClaudeBot collects content that could contribute to training, Claude-SearchBot reads pages to improve Claude's search results, and Claude-User fetches pages when someone asks Claude a question. Anthropic says blocking either of the last two may reduce a site's visibility in Claude's search results.

Perplexity says PerplexityBot surfaces and links websites in its search results and is not used to train foundation models, while Perplexity-User fetches pages for a user's question and generally ignores robots.txt.

Google runs no separate crawler for Gemini training or AI search. Google-Extended is a robots.txt token that Google reads as a control: it decides whether content Google already crawls may train future Gemini models and ground answers in Gemini Apps. Google says it does not affect a site's inclusion or ranking in Google Search.

What it means for your business

If you want your business named and linked in ChatGPT, Claude or Perplexity answers, their search bots need to reach your pages. Training is a separate choice. OpenAI and Anthropic name their training and search bots separately, so a business can opt out of training and stay in AI search.

For Google, there is nothing extra to allow. Blocking Google-Extended limits how Google may use your content to train and ground Gemini, and Google says it does not affect inclusion or ranking in Google Search. To keep a site out of AI Overviews and AI Mode, Google offers a separate Search generative AI control in Search Console, which includes every site by default. Check your hosting and firewall settings as well as robots.txt, because a bot blocked at the server never reads your rules. Getting these settings right is part of our AI search optimization work.

Common mistakes

• Blocking every AI bot with one rule to stop training. That also removes the site from AI search answers; block only the training bots if training is the concern.
• Assuming robots.txt stops every visit. OpenAI and Perplexity both say their user-triggered fetchers may not follow it, because a person asked for the page.
• Blocking Google-Extended to keep a site out of AI Overviews. Google's tool for that is the Search generative AI control in Search Console. Google says the token has no effect on Google Search.
• Copying bot names from unofficial lists. A misspelled user agent matches nothing, so use the names in each company's own documentation.

Related terms

Sources: OpenAI, Overview of OpenAI crawlers; Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?, 7 April 2026; Google Search Central, List of Google's common crawlers, 30 October 2026; Perplexity, Perplexity crawlers; Search Console Help, Search generative AI control.