AI Crawl Radar
Fix guides 5 min read

How to Fix AI Bot Access: Let GPTBot and ClaudeBot In

AI Bot Access decides whether GPTBot, ClaudeBot and PerplexityBot can read your site. Find what blocks them in robots.txt, a CDN or firewall, and fix it.

Cover image: How to Fix AI Bot Access: Let GPTBot and ClaudeBot In
On this page
  1. What the AI Bot Access check measures
  2. How the matching works
  3. The four common causes — and the fix for each
  4. Training bots vs. answer bots: a sensible default
  5. How to verify the fix
  6. Related guides

AI Bot Access is the single biggest part of your AI Crawl Radar score: it is worth 25 of the 100 points. It answers one question: when GPTBot, ClaudeBot, PerplexityBot and the other AI crawlers ask for your page, are they allowed in? If they are not, most of the other checks matter far less, because an AI system is far less likely to quote, summarize or cite a page it was never allowed to read.

This guide explains exactly how the check works, why a site can fail it without anyone deciding to block AI, and how to fix each cause.

What the AI Bot Access check measures

The audit reads your robots.txt and works out, for each of 21 crawlers, whether that crawler may fetch the page you audited and your homepage. Each bot gets one of three results:

  • Allowed — the rules that apply to this bot permit the page.
  • Blocked — a Disallow rule that applies to this bot covers the page.
  • Partial — the bot may fetch your homepage but not the audited page, or the other way around.

The bots are not weighted equally. Crawlers that fetch pages to answer a user's question right now — AI search and assistant bots such as OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User — count three times. AI training bots such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended count twice. Classic search engines and data aggregators count once. An allowed bot earns its full weight, a partial one half and a blocked one nothing. If you have no robots.txt at all, every bot is treated as allowed — crawling is permitted by default.

Why the weighting? Blocking a training crawler keeps your content out of future model training. Blocking a search or assistant crawler can keep you out of AI answers today. The second costs you visibility immediately, so it costs more points.

How the matching works

The audit follows the same rules Google documents for robots.txt:

  • A bot uses the group whose User-agent line names it most specifically. User-agent: GPTBot beats User-agent: *.
  • Only the product token at the start of the line counts, so User-agent: GPTBot/1.0 still means GPTBot.
  • A more specific token does not capture a shorter one: a group for Applebot-Extended (Apple's AI-training opt-out) does not apply to Applebot (Apple's search crawler).
  • Inside a group, the longest matching path wins; on a tie, Allow beats Disallow.

That matters when you read your result: if your report says GPTBot is blocked, there is a specific line in your robots.txt that does it, and the report quotes that line as the reason. On WordPress, the AI Visibility Optimizer plugin lets you set Allow or Disallow for each AI crawler from wp-admin.

Diagram: an AI crawler request passes a CDN or firewall, then robots.txt, before reaching the page; either can block it
Two gates stand between an AI crawler and your page.

The four common causes — and the fix for each

1. An explicit block in robots.txt

Someone added a group like this, often from a "block AI scrapers" guide:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Fix: decide which bots you want, then remove the block or narrow it. To let a bot in, delete its group or change it to:

User-agent: GPTBot
Allow: /

If you only want to keep AI crawlers out of private areas, block those paths instead of the whole site:

User-agent: GPTBot
Disallow: /account/
Disallow: /checkout/

2. A CDN or host that edits robots.txt for you

Some CDNs offer a managed robots.txt that prepends AI-crawler blocks to your own file. Cloudflare's "managed robots.txt" option, for example, adds Disallow: / groups for training crawlers such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. Your own file looks fine in your code repository, yet the robots.txt the world receives is different.

Fix: always check the live file at https://yourdomain.com/robots.txt, not the one in your project. If a CDN setting adds the blocks, turn that setting off or adjust it in the CDN dashboard.

3. A firewall that blocks the bot before robots.txt matters

A robots.txt file is a polite request. Firewalls and bot-protection services act on the request itself: they can return a 403 or a challenge page to anything that looks like a crawler. A bot can be "Allowed" in robots.txt and still never see your content. When the audit itself is challenged, it reports a Bot challenge detected finding in the HTTP Health & Performance category — see our guide to HTTP health and performance.

Fix: in your firewall or CDN, allowlist the verified AI crawlers you want rather than blocking every unknown bot. Many providers offer a "verified bots" category, and the major AI companies publish their crawlers' IP ranges, for example OpenAI and Perplexity.

4. A blanket block for every crawler

User-agent: *
Disallow: /

This is common on staging sites and is sometimes copied to production by mistake. It blocks every crawler without its own group, AI and search alike, and it also costs points in the robots.txt Strategy check.

Fix: replace it with Allow: / plus targeted Disallow lines for the areas that really are private.

Report table listing each AI crawler with its provider, access status, risk and the robots.txt rule behind the verdict
The AI & search crawler access table in a report quotes the rule that blocks each crawler.

Training bots vs. answer bots: a sensible default

You do not have to choose between "everything open" and "everything closed". Many sites keep training crawlers out but let search and assistant crawlers in, so they appear in AI answers without contributing to model training:

User-agent: *
Allow: /

# Out of model training
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# In AI search and assistant answers
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

With this file, GPTBot and Google-Extended score zero and the search bots score full marks, so AI Bot Access still scores about 23 of 25. Expect two side effects: robots.txt Strategy loses its 3-point "no homepage block" share, and the report shows a "robots.txt and meta robots disagree" warning. The audit flags every deliberate block, so read those as confirmation of your choice, not as errors. Whichever policy you choose, make it a decision rather than an accident.

Tip: Google-Extended is not Googlebot. Blocking Google-Extended controls whether your content is used for Gemini training and grounding; it does not affect Google Search. Blocking Googlebot stops Google crawling your pages, which in practice drops them from Google Search. See Google's list of its crawlers.

How to verify the fix

  1. Open https://yourdomain.com/robots.txt in a private browser window and confirm the live file contains your new rules.
  2. Run a fresh audit in the AI visibility audit tool. Each bot's status is listed in the AI & search crawler access table; for a blocked bot, the reason quotes the rule that blocks it.
  3. If you use a firewall, check its logs for requests from the bot's user-agent and confirm they return 200, not 403.

If your report is public on AI Crawl Radar, the re-check button on the report page refreshes it. Reports also refresh automatically once they are older than 30 days.

Frequently asked questions

6 questions people ask about this topic

Should I block GPTBot?
Only if you don't want your content used to train OpenAI's models. OpenAI documents that its crawler settings are independent: blocking GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot, and ChatGPT-User fetches pages when a user asks.
If I block GPTBot, will my site disappear from ChatGPT?
No. ChatGPT search results come from OAI-SearchBot. To stay out of ChatGPT search too, you have to disallow OAI-SearchBot as well. OpenAI says robots.txt changes for search can take about 24 hours to take effect.
Which Anthropic crawlers should I allow for Claude?
Anthropic runs three: ClaudeBot (collects content that may be used for training), Claude-SearchBot (indexes pages for search) and Claude-User (fetches pages when a Claude user asks). Blocking ClaudeBot alone does not block the other two; each needs its own robots.txt group.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Google Search uses Googlebot, which Google-Extended rules do not affect.
Why are AI crawlers blocked when my robots.txt allows them?
Usually a CDN or firewall is the cause. Cloudflare's managed robots.txt can prepend Disallow rules to your file, and bot protection can return a 403 or a challenge before robots.txt matters. Check the live /robots.txt, test with a crawler user agent, and look at your firewall events.
Do AI crawlers actually respect robots.txt?
The major declared crawlers say they do, and OpenAI, Anthropic and Google document robots.txt support. User-triggered fetchers such as ChatGPT-User may not apply robots.txt rules, because a person requested the page. Enforcement is another matter: Cloudflare has reported a crawler evading blocks with undeclared user agents. For content that must stay private, use authentication, not robots.txt.

See how AI crawlers read your site

Run a free AI visibility audit: a 0–100 score and a fix for every lost point.

Analyze my website

Keep reading

All guides →