AI Crawl Radar
Fix guides 4 min read

How to Optimize robots.txt for AI Crawlers

robots.txt for AI crawlers: fix a missing file, undeclared sitemaps and syntax errors, and learn the rules that decide what GPTBot and ClaudeBot may crawl.

Cover image: How to Optimize robots.txt for AI Crawlers
On this page
  1. How the robots.txt Strategy score is calculated
  2. The findings and fixes you may see
  3. Rules that trip people up
  4. A solid starting template
  5. Verify
  6. Related guides

Your robots.txt is the first file most crawlers request, and getting robots.txt for AI crawlers right decides what GPTBot, ClaudeBot and others may fetch. It tells them which parts of your site they may fetch and where your sitemap lives. In an AI Crawl Radar audit it drives two categories: AI Bot Access, which asks whether each AI crawler is let in, and robots.txt Strategy, worth 15 of the 100 points, which asks whether the file itself is well built. This guide covers the second.

How the robots.txt Strategy score is calculated

The score is built from four signals:

SignalShare of the category
A robots.txt file exists50% (40% if it is missing — crawling is still allowed by default)
It declares at least one Sitemap:+30%
None of the 21 tracked crawlers is blocked from your homepage+20%
Syntax warnings−5% each, up to −20%

A file that exists, lists your sitemap, blocks nobody from the homepage and parses cleanly scores the full 15 points.

The findings and fixes you may see

"No robots.txt found"

Without a robots.txt everything is crawlable, which is fine for access but wastes the file's other job: pointing crawlers at your sitemap. The report suggests a minimal file. Put it at the root of your domain, so it is served at https://yourdomain.com/robots.txt:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

It must be at the root. A robots.txt in a subfolder is ignored, and each subdomain needs its own file.

"Declare your sitemap in robots.txt"

Your file exists but has no Sitemap: line. Add one with the full absolute URL. The line can go anywhere in the file and does not belong to any user-agent group:

Sitemap: https://yourdomain.com/sitemap.xml

If you have several sitemaps, list each one, or list the sitemap index file. More on sitemaps in Sitemaps and AI discovery.

"robots.txt has syntax warnings"

The audit reports the line numbers it could not use. The two warnings it raises are:

  • Missing ':' separator — every directive needs a colon, as in Disallow: /admin/. Many parsers ignore a line such as Disallow /admin/. Google tolerates it, but you cannot count on other crawlers doing the same, so the rule you thought you had may not exist.
  • Allow or Disallow before any User-agent — rules must come after the User-agent line they belong to. Rules at the top of the file with no group above them apply to nobody.

Fix the reported lines, then reload the live file to make sure your edit is what is actually served.

Homepage blocked for a crawler

If any crawler is disallowed from /, the category loses its 20% "no homepage block" share, and the bot in question is also marked Blocked under AI Bot Access (or Partial, if only the homepage is blocked). Sometimes that is exactly your intent, for example blocking a training crawler. Sometimes it is a leftover Disallow: / from a staging site. Check every group with Disallow: / and keep only the ones you meant.

Rules that trip people up

Specific groups replace the general one

A crawler obeys only the most specific group that names it. If you write:

User-agent: *
Disallow: /private/

User-agent: GPTBot
Disallow: /drafts/

GPTBot follows only its own group. It may crawl /private/, because the * rules no longer apply to it. Repeat any shared rules inside each named group.

Example robots.txt with three groups and the group each crawler follows: GPTBot and Applebot-Extended their own, Applebot and ClaudeBot the * group
Each crawler follows the most specific group that names it.

The longest matching path wins

Within a group, the rule with the longest matching path takes effect, and Allow wins a tie. This lets you open one folder inside a blocked area:

User-agent: *
Disallow: /docs/
Allow: /docs/public/

Wildcards

* matches any sequence of characters and $ anchors the end of the URL. Disallow: /*.pdf$ blocks PDFs only.

Check the live file, not the file in your code

CDNs and some hosting panels can add rules to robots.txt on the way out. The file that counts is the one served at your domain, which is also the one the audit reads. Fixing AI bot access covers the Cloudflare case.

Remember: robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it. To keep a page out of results, let it be crawled and add a noindex meta tag. Our technical SEO guide covers the conflict this creates.

A solid starting template

# Everyone may crawl the public site
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/

Sitemap: https://yourdomain.com/sitemap.xml

From here, add named groups only for crawlers you want to treat differently. Fixing AI bot access shows an example that keeps training bots out but lets AI search bots in.

Verify

  1. Open /robots.txt on your live domain and read it top to bottom.
  2. Re-run the AI visibility audit. The robots.txt Strategy category shows the new score, and AI Bot Access shows the blocking rule for any blocked bot.
  3. On WordPress, the AI Visibility Optimizer plugin has a robots.txt assistant that merges AI-friendly rules with your SEO plugin's rules instead of overwriting them.

Frequently asked questions

6 questions people ask about this topic

Where does robots.txt have to be?
At the root of the host, served at https://yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored, and each subdomain needs its own file.
Does robots.txt keep a page out of search results?
No. robots.txt controls crawling, not indexing, and a blocked URL can still appear if other sites link to it. To keep a page out of results, allow crawling and add a noindex meta tag.
Do I need a separate group for each AI crawler?
Only for crawlers you want to treat differently. A named group replaces the User-agent: * group for that bot, so repeat any shared rules inside it.
Can I block AI crawlers from some pages but not others?
Yes. Put Disallow lines for the private paths, such as /account/ or /checkout/, inside the crawler's group and leave everything else allowed. The longest matching rule wins, so an Allow for a subfolder can reopen part of a blocked area.
Is Crawl-delay supported?
Not by Google, which ignores it. Some crawlers do honor it: Anthropic says it supports Crawl-delay for slowing its crawlers instead of blocking them. If server load is the concern, rate limiting at your CDN is more reliable.
How do I test my robots.txt?
Open the live file in a browser, not the copy in your code, because CDNs can change it on the way out. Then run an audit: the report shows, for each of 21 crawlers, whether it is allowed and which rule blocks it.

See how AI crawlers read your site

Run a free AI visibility audit: a 0–100 score and a fix for every lost point.

Analyze my website

Keep reading

All guides →