AI Crawl Radar
AI crawler guides 5 min read

AI Crawler List: Every Major AI Bot, Its User Agent and What It Does

Who runs each AI bot, whether it trains models, powers AI search or fetches pages for users, and the exact robots.txt token to use.

AI Crawler List cover: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, ChatGPT-User and Google-Extended with their roles
On this page
  1. The three kinds of AI crawler
  2. The AI crawler list
  3. Names you will still see in old robots.txt files
  4. Which AI crawlers follow robots.txt?
  5. How to see which AI crawlers can reach your site
  6. How the audit weighs each kind
  7. Related guides

Dozens of bots now visit websites on behalf of AI companies, and they do not all want the same thing. Some collect pages to train future models, some build the index an AI search engine answers from, and some fetch a single page because a person just asked a chatbot about it. If you are deciding what to allow in robots.txt, reading your server logs, or trying to understand an AI visibility report, you need to know which is which.

This AI crawler list covers every major AI bot as of 2026: who runs it, what it is for, and the exact token to use in robots.txt. It covers the crawlers the AI Crawl Radar audit checks, plus the newer bots listed in each operator's current documentation, which is linked at the end.

The three kinds of AI crawler

Almost every AI bot does one of three jobs. The job matters more than the company name, because blocking each kind costs you something different.

  • Training crawlers collect pages that may be used to train future AI models. Examples: GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot. Blocking them keeps future content out of training data. It does not remove you from AI search answers.
  • AI search crawlers build the index an AI answer engine searches when someone asks a question. Examples: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot. Blocking them is the surest way to disappear from those answers and their citations.
  • User-triggered fetchers load one page, right now, because a user asked about it or pasted the link. Examples: ChatGPT-User, Claude-User, Perplexity-User. They do not crawl your site; they act on a single request.
Diagram of the three kinds of AI crawler: training crawlers, AI search crawlers and user-triggered fetchers, with example bots from OpenAI, Anthropic, Perplexity, Google and Meta
The same company usually runs one bot for each job, and each can be allowed or blocked on its own.

The AI crawler list

The first column is the robots.txt token: the name you write after User-agent:. Tokens are not case-sensitive.

TokenOperatorJobWhat it is for
GPTBotOpenAITrainingCollects content that may be used to train OpenAI's models.
OAI-SearchBotOpenAIAI searchSurfaces sites in ChatGPT search answers and citations.
ChatGPT-UserOpenAIUser fetchVisits a page when a ChatGPT or Custom GPT user asks about it.
ClaudeBotAnthropicTrainingCollects content that could contribute to training Claude models.
Claude-SearchBotAnthropicAI searchIndexes content to improve Claude's search results.
Claude-UserAnthropicUser fetchRetrieves a page when a Claude user's question needs it.
PerplexityBotPerplexityAI searchSurfaces and links sites in Perplexity answers. Not used for model training.
Perplexity-UserPerplexityUser fetchVisits a page to help answer a user's question.
GooglebotGoogleSearchGoogle Search, including the AI features built into Search.
Google-ExtendedGoogleTraining controlA control token for Gemini training and grounding. It has no crawler of its own.
BingbotMicrosoftSearchBing Search, whose index also feeds Microsoft's Copilot answers.
ApplebotAppleSearchSiri and Spotlight suggestions.
Applebot-ExtendedAppleTraining controlA control token for Apple's generative AI training. It does not crawl.
Meta-ExternalAgentMetaTrainingCrawls for Meta's AI products, such as Llama and Meta AI.
Meta-ExternalFetcherMetaUser fetchFetches a link on behalf of a Meta AI user.
AmazonbotAmazonAI answersFeeds Alexa and Amazon's AI answers.
DuckAssistBotDuckDuckGoUser fetchPowers DuckDuckGo's AI-assisted answers.
CCBotCommon CrawlDatasetBuilds the open Common Crawl archive that many AI training datasets are drawn from.
BytespiderByteDanceTrainingByteDance's crawler for its AI models. Many sites choose to block it.
Diffbot, OmgilibotDiffbot, Webz.ioDataExtract and resell web data that ends up in knowledge graphs and AI datasets.

Two tokens on the list never visit you. Google-Extended and Applebot-Extended are opt-out switches, not crawlers. Google says crawling is done with its existing user agents and the Google-Extended token "is used in a control capacity". You will never see either in your logs, but a Disallow for them is still read and respected.

Names you will still see in old robots.txt files

Bots get renamed. Many robots.txt files still carry tokens such as anthropic-ai and Claude-Web from Anthropic's earlier naming. Anthropic's current documentation lists three bots: ClaudeBot, Claude-User and Claude-SearchBot. Keeping an old token does no harm, but a rule written only for an old name does not cover the new bot, so check that your rules name the current tokens.

Which AI crawlers follow robots.txt?

Crawlers that work through your site on a schedule (training and AI search bots) follow robots.txt according to their operators. User-triggered fetchers are the exception to watch:

  • OpenAI says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply."
  • Perplexity says Perplexity-User "generally ignores robots.txt rules", because a user asked for the page.
  • Anthropic says all three of its bots honor robots.txt, and ClaudeBot also supports the non-standard Crawl-delay directive.

So robots.txt reliably controls crawling and training, but it is not a dependable way to stop a chatbot reading a page a user has pointed it at. If a page must never be read, put it behind a login.

How to see which AI crawlers can reach your site

There are three practical ways, depending on what you want to know.

  1. Which bots your robots.txt allows. Run a free AI visibility audit. The report lists each crawler it checks with its status (Allowed, Blocked, Partial or Unknown) and, for a blocked bot, the rule that blocks it.
  2. Which bots actually visit. On WordPress, the free AI Visibility Optimizer plugin logs AI crawler visits and labels each one by job and by whether it is a genuine crawler. See the plugin setup guide.
  3. Your raw server logs. Search the access log for the tokens above, for example grep -iE "gptbot|oai-searchbot|claudebot|perplexitybot" access.log. Remember that a user agent can be faked; verifying AI crawlers explains how to tell real bots from impostors.
Report table listing each AI crawler with its provider, access status, risk and the robots.txt rule behind the verdict
The audit report shows each crawler's access and the rule behind it.

How the audit weighs each kind

AI Bot Access is worth 25 of the 100 points in an AI Crawl Radar score, and the bots are not weighted equally. AI search and user-fetch bots count three times, because blocking them removes you from answers today. Training bots count twice. Classic search engines and data aggregators count once. That is why a site that blocks only training bots can still score well, while one that blocks OAI-SearchBot or PerplexityBot loses points fast. Should you block AI crawlers? walks through the trade-off.

Frequently asked questions

6 questions people ask about this topic

What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot indexes pages so they can appear and be cited in ChatGPT search. They are controlled separately, so you can block GPTBot and still allow OAI-SearchBot.
Which AI crawlers should I allow?
If you want to be cited in AI answers, allow the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and keep Googlebot and Bingbot allowed. Training crawlers such as GPTBot and ClaudeBot are a separate choice that does not affect AI search citations.
Why don't I see Google-Extended in my server logs?
Google-Extended is not a crawler. It is a robots.txt token that controls whether content Google crawls can be used for Gemini training and grounding. Google's existing crawlers do the fetching, so the token never appears as a user agent.
Do AI crawlers respect robots.txt?
Training and AI search crawlers from OpenAI, Anthropic, Perplexity and Google follow robots.txt according to their documentation. User-triggered fetchers are different: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.
Are anthropic-ai and Claude-Web still used?
They are older Anthropic tokens that still appear in many robots.txt files. Anthropic's current documentation lists ClaudeBot, Claude-User and Claude-SearchBot, so make sure your rules name those.
How can I check which AI crawlers can access my site?
Run a free AI visibility audit. The report lists each major AI crawler with its status (Allowed, Blocked, Partial or Unknown) and, for a blocked bot, the robots.txt rule responsible.

See how AI crawlers read your site

Run a free AI visibility audit: a 0–100 score and a fix for every lost point.

Analyze my website

Keep reading

All guides →