On this page
Dozens of bots now visit websites on behalf of AI companies, and they do not all want the same thing. Some collect pages to train future models, some build the index an AI search engine answers from, and some fetch a single page because a person just asked a chatbot about it. If you are deciding what to allow in robots.txt, reading your server logs, or trying to understand an AI visibility report, you need to know which is which.
This AI crawler list covers every major AI bot as of 2026: who runs it, what it is for, and the exact token to use in robots.txt. It covers the crawlers the AI Crawl Radar audit checks, plus the newer bots listed in each operator's current documentation, which is linked at the end.
The three kinds of AI crawler
Almost every AI bot does one of three jobs. The job matters more than the company name, because blocking each kind costs you something different.
- Training crawlers collect pages that may be used to train future AI models. Examples: GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot. Blocking them keeps future content out of training data. It does not remove you from AI search answers.
- AI search crawlers build the index an AI answer engine searches when someone asks a question. Examples: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot. Blocking them is the surest way to disappear from those answers and their citations.
- User-triggered fetchers load one page, right now, because a user asked about it or pasted the link. Examples: ChatGPT-User, Claude-User, Perplexity-User. They do not crawl your site; they act on a single request.

The AI crawler list
The first column is the robots.txt token: the name you write after User-agent:. Tokens are not case-sensitive.
| Token | Operator | Job | What it is for |
|---|---|---|---|
GPTBot | OpenAI | Training | Collects content that may be used to train OpenAI's models. |
OAI-SearchBot | OpenAI | AI search | Surfaces sites in ChatGPT search answers and citations. |
ChatGPT-User | OpenAI | User fetch | Visits a page when a ChatGPT or Custom GPT user asks about it. |
ClaudeBot | Anthropic | Training | Collects content that could contribute to training Claude models. |
Claude-SearchBot | Anthropic | AI search | Indexes content to improve Claude's search results. |
Claude-User | Anthropic | User fetch | Retrieves a page when a Claude user's question needs it. |
PerplexityBot | Perplexity | AI search | Surfaces and links sites in Perplexity answers. Not used for model training. |
Perplexity-User | Perplexity | User fetch | Visits a page to help answer a user's question. |
Googlebot | Search | Google Search, including the AI features built into Search. | |
Google-Extended | Training control | A control token for Gemini training and grounding. It has no crawler of its own. | |
Bingbot | Microsoft | Search | Bing Search, whose index also feeds Microsoft's Copilot answers. |
Applebot | Apple | Search | Siri and Spotlight suggestions. |
Applebot-Extended | Apple | Training control | A control token for Apple's generative AI training. It does not crawl. |
Meta-ExternalAgent | Meta | Training | Crawls for Meta's AI products, such as Llama and Meta AI. |
Meta-ExternalFetcher | Meta | User fetch | Fetches a link on behalf of a Meta AI user. |
Amazonbot | Amazon | AI answers | Feeds Alexa and Amazon's AI answers. |
DuckAssistBot | DuckDuckGo | User fetch | Powers DuckDuckGo's AI-assisted answers. |
CCBot | Common Crawl | Dataset | Builds the open Common Crawl archive that many AI training datasets are drawn from. |
Bytespider | ByteDance | Training | ByteDance's crawler for its AI models. Many sites choose to block it. |
Diffbot, Omgilibot | Diffbot, Webz.io | Data | Extract and resell web data that ends up in knowledge graphs and AI datasets. |
Two tokens on the list never visit you. Google-Extended and Applebot-Extended are opt-out switches, not crawlers. Google says crawling is done with its existing user agents and the Google-Extended token "is used in a control capacity". You will never see either in your logs, but a Disallow for them is still read and respected.
Names you will still see in old robots.txt files
Bots get renamed. Many robots.txt files still carry tokens such as anthropic-ai and Claude-Web from Anthropic's earlier naming. Anthropic's current documentation lists three bots: ClaudeBot, Claude-User and Claude-SearchBot. Keeping an old token does no harm, but a rule written only for an old name does not cover the new bot, so check that your rules name the current tokens.
Which AI crawlers follow robots.txt?
Crawlers that work through your site on a schedule (training and AI search bots) follow robots.txt according to their operators. User-triggered fetchers are the exception to watch:
- OpenAI says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply."
- Perplexity says Perplexity-User "generally ignores robots.txt rules", because a user asked for the page.
- Anthropic says all three of its bots honor robots.txt, and ClaudeBot also supports the non-standard
Crawl-delaydirective.
So robots.txt reliably controls crawling and training, but it is not a dependable way to stop a chatbot reading a page a user has pointed it at. If a page must never be read, put it behind a login.
How to see which AI crawlers can reach your site
There are three practical ways, depending on what you want to know.
- Which bots your robots.txt allows. Run a free AI visibility audit. The report lists each crawler it checks with its status (Allowed, Blocked, Partial or Unknown) and, for a blocked bot, the rule that blocks it.
- Which bots actually visit. On WordPress, the free AI Visibility Optimizer plugin logs AI crawler visits and labels each one by job and by whether it is a genuine crawler. See the plugin setup guide.
- Your raw server logs. Search the access log for the tokens above, for example
grep -iE "gptbot|oai-searchbot|claudebot|perplexitybot" access.log. Remember that a user agent can be faked; verifying AI crawlers explains how to tell real bots from impostors.

How the audit weighs each kind
AI Bot Access is worth 25 of the 100 points in an AI Crawl Radar score, and the bots are not weighted equally. AI search and user-fetch bots count three times, because blocking them removes you from answers today. Training bots count twice. Classic search engines and data aggregators count once. That is why a site that blocks only training bots can still score well, while one that blocks OAI-SearchBot or PerplexityBot loses points fast. Should you block AI crawlers? walks through the trade-off.
Related guides
Frequently asked questions
6 questions people ask about this topic
What is the difference between GPTBot and OAI-SearchBot?
Which AI crawlers should I allow?
Why don't I see Google-Extended in my server logs?
Do AI crawlers respect robots.txt?
Are anthropic-ai and Claude-Web still used?
How can I check which AI crawlers can access my site?
See how AI crawlers read your site
Run a free AI visibility audit: a 0–100 score and a fix for every lost point.
Analyze my website


