AI Crawl Radar
Fix guides 3 min read

XML Sitemaps for AI Crawlers: Get Every Page Discovered

XML sitemaps for AI crawlers help them find every page. How the audit checks yours, where it lives on WordPress and Next.js, and how to declare it.

Cover image: XML Sitemaps for AI Crawlers: Get Every Page Discovered
On this page
  1. How the check works
  2. "Publish an XML sitemap"
  3. "Declare your sitemap in robots.txt"
  4. Missing lastmod dates
  5. Do AI crawlers use sitemaps?
  6. What belongs in a sitemap
  7. Sitemaps and llms.txt
  8. Verify
  9. Related guides

Crawlers find pages in two ways: by following links, and by reading a list you give them. That list is your XML sitemap, and sitemaps for AI crawlers work the same way they do for search engines. It matters most for pages that are new, deep in your site or poorly linked, which are exactly the pages a crawler following links might never reach.

Sitemap & Discovery is worth 5 of the 100 points in an AI Crawl Radar audit. It is one of the easiest categories to score in full.

How the check works

The audit requests https://yourdomain.com/sitemap.xml (following redirects) and reads your robots.txt. Then:

SignalShare of the category
A sitemap is served at /sitemap.xml with a 2xx status60%
robots.txt declares a sitemap with a Sitemap: line25%
The sitemap includes <lastmod> dates15%

A sitemap index (a sitemap that lists other sitemaps) counts as a sitemap.

Diagram: robots.txt declares the sitemap, the sitemap lists every page; with the category's 60, 25 and 15 percent shares
How crawlers discover pages, and how the audit scores it.

"Publish an XML sitemap"

This recommendation appears when nothing usable is served at /sitemap.xml. A minimal sitemap looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yourdomain.com/</loc>
    <lastmod>2026-10-01</lastmod>
  </url>
  <url>
    <loc>https://yourdomain.com/pricing</loc>
    <lastmod>2026-09-14</lastmod>
  </url>
</urlset>

You rarely need to write it by hand:

  • WordPress: WordPress has a built-in sitemap at /wp-sitemap.xml, and SEO plugins such as Yoast and Rank Math generate their own, usually at /sitemap_index.xml.
  • Next.js: add an app/sitemap.ts file that returns your URLs; Next serves it at /sitemap.xml.
  • Shopify, Wix, Squarespace: a sitemap is generated automatically at /sitemap.xml.
  • Other stacks: most frameworks and static site generators have a sitemap plugin.

When your sitemap lives somewhere else

The audit looks at /sitemap.xml specifically. If your sitemap is only at /sitemap_index.xml or /wp-sitemap.xml, add a 301 redirect from /sitemap.xml to it. The audit follows the redirect, /sitemap.xml is the conventional location crawlers check, and it costs nothing. Many SEO plugins already add this redirect.

"Declare your sitemap in robots.txt"

Add one line anywhere in your robots.txt, using the full URL:

Sitemap: https://yourdomain.com/sitemap.xml

This is how crawlers that have never heard of your site find your sitemap: they read robots.txt first. The line is not part of any user-agent group, so it applies to every crawler. See the robots.txt guide for the rest of the file.

Missing lastmod dates

<lastmod> tells crawlers when a page last changed, so they can re-fetch what is new instead of everything. Two rules make it useful:

  • Make it true. Use the date the content actually changed. A sitemap generator that stamps every URL with today's date on every request teaches crawlers to ignore your dates.
  • Use W3C format: 2026-10-01 or 2026-10-01T09:30:00+00:00.

Do AI crawlers use sitemaps?

AI providers rarely document it, but crawlers commonly read the Sitemap: line in robots.txt, and a sitemap also feeds the search indexes many AI answers draw on. It costs nothing to publish.

What belongs in a sitemap

  • Include the canonical, indexable version of every page you want found.
  • Exclude pages with noindex (see technical SEO for AI visibility), pages blocked in robots.txt, redirects, 404s, duplicate URLs with tracking parameters, and private areas such as account and checkout pages.
  • Keep each file under 50,000 URLs and 50 MB uncompressed; use a sitemap index to split larger sites.

Submit it, too. Add your sitemap in Google Search Console and Bing Webmaster Tools. For Bing, Yandex and others you can also notify them of new and changed URLs as soon as you publish, with IndexNow.

Sitemaps and llms.txt

A sitemap and an llms.txt do different jobs. The sitemap is a complete list for crawlers; llms.txt is a short, described shortlist for AI tools. Have both: the sitemap so nothing is missed, llms.txt so the important pages are understood.

Verify

  1. Open https://yourdomain.com/sitemap.xml and confirm it loads as XML and lists your key pages.
  2. Check robots.txt for the Sitemap: line.
  3. Re-run the AI visibility audit; Sitemap & Discovery should reach 100%.

Frequently asked questions

6 questions people ask about this topic

Do lastmod dates need to be accurate?
Yes. Use the date the content actually changed. A generator that stamps every URL with today's date on each request teaches crawlers to ignore your dates.
Where is my WordPress sitemap?
WordPress serves one at /wp-sitemap.xml, and Yoast or Rank Math usually use /sitemap_index.xml. If /sitemap.xml returns nothing, redirect it to your real sitemap.
Do I need both a sitemap and llms.txt?
They do different jobs. The sitemap is a complete list for crawlers; llms.txt is an optional short list of key pages for AI tools. The sitemap is the one that matters for discovery.
Should I submit my sitemap to Bing?
Yes. Submit it in Bing Webmaster Tools as well as Google Search Console. Bing's index feeds several AI search tools, and you can also notify Bing instantly with IndexNow.
What should not be in a sitemap?
Redirects, 404s, noindexed pages, URLs blocked in robots.txt, and duplicate URLs with tracking parameters. Every listed URL should be a canonical page you want found.
How big can a sitemap be?
Up to 50,000 URLs and 50 MB uncompressed per file. Larger sites split their URLs across several sitemaps listed in a sitemap index.

See how AI crawlers read your site

Run a free AI visibility audit: a 0–100 score and a fix for every lost point.

Analyze my website

Keep reading

All guides →