AI Crawl Radar
How-to guides 5 min read

What AI Crawlers See on Your Page: Inspect the Raw HTML and JSON-LD

Most AI crawlers read one HTTP response and never run your JavaScript. Here is how to see that response for yourself, and what to fix in it.

What AI Crawlers See on Your Page cover with a BlogPosting JSON-LD block
On this page
  1. What a crawler actually reads
  2. Three ways to see it for yourself
  3. How to read the result
  4. Check that the JSON-LD matches the page
  5. Let the audit do it for every check at once
  6. Related guides

Every web page has two layers. Visitors see the design, the images and the text once the browser has finished running its scripts. Crawlers read something else: the raw HTML your server sends back, plus the machine-readable layer inside it, such as the title tag, meta tags, canonical link and JSON-LD structured data. When the two layers disagree, crawlers see the second one.

That matters more for AI crawlers than for Google. A December 2024 study by Vercel found that the major AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, do not run JavaScript, while Googlebot (and Gemini, which uses its infrastructure) and Applebot do. Whatever is not in the first HTML response is, for most AI crawlers, simply not there. This guide shows how to see your page the way they do, in a few minutes, with tools you already have.

What a crawler actually reads

When GPTBot or ClaudeBot requests a page, it gets one HTTP response and works from that. The parts that carry meaning are:

  • The status code and headers, including any X-Robots-Tag header that tells crawlers not to index the page.
  • The title and meta description, the shortest statement of what the page is about.
  • The robots meta tag and canonical link, which say whether the page may be indexed and which URL is the original.
  • JSON-LD structured data, which states facts outright: this is an article, by this author, published on this date, for this organization.
  • The text and links in the raw HTML, which is all the content a non-rendering crawler gets.
Diagram comparing the visitor's view of a page, built after JavaScript runs, with the crawler's view: status and headers, title and meta, canonical and robots, JSON-LD and the text in the raw HTML
Visitors see the rendered page. Most AI crawlers only see the raw response.

Three ways to see it for yourself

1. View the source, not the inspector

In your browser, open the page and press Ctrl+U (or Cmd+Option+U on a Mac). That shows the HTML exactly as the server sent it. The developer tools' Elements panel shows the page after JavaScript has changed it, which is what Google eventually renders but not what most AI crawlers read. Search the source for application/ld+json, <title and a sentence from your main content. If the sentence is missing, the content is added by JavaScript.

2. Fetch it as the crawler

Some servers, CDNs and security plugins answer bots differently from browsers. Request the page with GPTBot's user agent and compare it with a normal request:

UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot"

curl -s -o /dev/null -w "browser: %{http_code}\n" https://www.example.com/
curl -s -o /dev/null -w "GPTBot:  %{http_code}\n" -A "$UA" https://www.example.com/
curl -s -A "$UA" https://www.example.com/ | less

A 403, a 429 or a challenge page for the GPTBot request where the browser gets 200 means something is treating AI crawlers differently. Read the result with care: your request does not come from OpenAI's IP addresses, so a firewall that verifies crawlers may block your test and still let the real GPTBot in. A block here is a lead to check in your firewall or CDN settings, not proof.

3. Summarize it with a script

The short Python script below fetches a page as GPTBot and prints what a crawler would take away: the title, the key meta tags, how many words are in the raw HTML, and every JSON-LD block with its type. It uses only the standard library, so it runs anywhere Python 3 is installed.

import json, re, sys, urllib.request
from html.parser import HTMLParser

UA = "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot"

class Page(HTMLParser):
    def __init__(self):
        super().__init__()
        self.meta, self.jsonld, self.text, self.title = {}, [], [], ""
        self._in = None
    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "meta" and a.get("content"):
            key = a.get("name") or a.get("property")
            if key:
                self.meta[key.lower()] = a["content"]
        elif tag == "link" and a.get("rel") == "canonical":
            self.meta["canonical"] = a.get("href")
        elif tag == "script" and a.get("type") == "application/ld+json":
            self._in = "jsonld"; self.jsonld.append("")
        elif tag in ("script", "style", "noscript"):
            self._in = "skip"
        elif tag == "title":
            self._in = "title"
    def handle_endtag(self, tag):
        if tag in ("script", "style", "noscript", "title"):
            self._in = None
    def handle_data(self, data):
        if self._in == "jsonld": self.jsonld[-1] += data
        elif self._in == "title": self.title += data
        elif self._in is None: self.text.append(data)

req = urllib.request.Request(sys.argv[1], headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=20) as r:
    print("HTTP", r.status, r.headers.get("content-type"))
    page = Page(); page.feed(r.read().decode("utf-8", "replace"))

print("Title:      ", page.title.strip())
for k in ("description", "robots", "canonical", "og:title"):
    print(f"{k + ':':12}", page.meta.get(k, "(missing)"))
print("Words in raw HTML:", len(re.findall(r"\w+", " ".join(page.text))))

print(f"\nJSON-LD blocks: {len(page.jsonld)}")
for i, block in enumerate(page.jsonld, 1):
    try:
        data = json.loads(block)
    except ValueError as e:
        print(f"  #{i} PARSE ERROR: {e}"); continue
    items = data if isinstance(data, list) else data.get("@graph", [data])
    print(f"  #{i}", ", ".join(str(x.get("@type")) for x in items if isinstance(x, dict)))

Save it as crawler_view.py and run python3 crawler_view.py https://www.example.com/your-page. Unlike a quick one-liner that stops at the first match, it lists every JSON-LD block, which is where most problems hide.

Terminal output of the crawler_view.py script for an article page: HTTP 200, title, description, robots, canonical, word count and five JSON-LD blocks
The script's output for one of our own articles: everything a crawler takes away, in ten lines.

How to read the result

What you seeWhat it meansWhat to do
Status other than 200The crawler gets an error, a redirect chain or a block instead of your page.See firewalls, redirects and bot challenges.
robots: noindexYou are asking search and AI systems not to index this page.Remove it unless the page really should stay out. See technical SEO for AI.
Description or canonical (missing)Crawlers have to guess the summary and the original URL.Add both. The canonical should usually point to the page itself.
Very few words in the raw HTMLThe visible content is rendered by JavaScript, so most AI crawlers miss it.Serve the main content in the HTML. See fixing JavaScript-only pages.
JSON-LD blocks: 0Nothing on the page states its type, author or organization.Add the recommended types from structured data for AI.
PARSE ERRORThe block is invalid JSON, so crawlers ignore all of it.Look for trailing commas, unescaped quotes or a template variable left in.
The same type twice with different valuesTwo plugins or templates are both emitting schema and contradict each other.Keep one source and switch the other off.

Watch for schema added by a tag manager. JSON-LD injected with Google Tag Manager or another script appears in the browser's inspector but not in the page source. Google can pick it up after rendering. Most AI crawlers never will. If the script above shows fewer JSON-LD blocks than your browser's inspector, that is the reason.

Check that the JSON-LD matches the page

Valid JSON-LD can still be wrong. For each block, check three things:

  1. The @type fits the page. An article should be Article or BlogPosting, a product page Product, and the homepage usually Organization and WebSite.
  2. The values match what visitors see. The headline, author, dates and prices in the markup should be the ones on the page. Markup that describes content visitors cannot see is against Google's guidelines and helps no one.
  3. The basics are filled in. An article without an author or a date, or an organization without a name and URL, leaves crawlers to infer exactly what structured data was meant to state.

For a second opinion, paste the URL into Google's Rich Results Test, which shows what Google can use after rendering, and the Schema Markup Validator, which checks against the full schema.org vocabulary.

Let the audit do it for every check at once

The AI Crawl Radar audit reads your page the same way: one request, raw HTML, no JavaScript. It then reports what it found in the same layers covered here, including the response and headers, title, description, canonical and robots tags, whether the content is available without JavaScript, how many JSON-LD blocks parse, and which recommended schema types are missing, with a ready-to-paste snippet for each. Running it after a change is the quickest way to confirm that the crawler's view of the page matches yours.

Frequently asked questions

6 questions people ask about this topic

Do AI crawlers run JavaScript?
Most do not. A December 2024 study by Vercel found that GPTBot, ClaudeBot and PerplexityBot fetch pages without running JavaScript, while Googlebot (which Gemini uses) and Applebot do render it. Content that only appears after scripts run is invisible to most AI crawlers.
How can I see my page the way GPTBot sees it?
View the page source with Ctrl+U rather than the browser inspector, or fetch the page with curl using GPTBot's user agent. Both show the raw HTML response, which is what non-rendering crawlers read.
Why does my JSON-LD show in the inspector but not in the page source?
It is being added by JavaScript, often through Google Tag Manager or a plugin script. Google can read it after rendering, but most AI crawlers never run that script, so move the JSON-LD into the HTML your server sends.
Does it matter if my page has several JSON-LD blocks?
Several blocks are normal, for example Organization, WebSite and Article. Problems start when two blocks describe the same thing with different values, usually because two plugins both output schema. Keep one source.
Is fetching my page with a GPTBot user agent a reliable test?
It shows how your server and CDN treat that user agent, which is useful. It is not conclusive, because the request does not come from OpenAI's IP addresses, so a firewall that verifies crawlers may treat your test differently from the real GPTBot.
Which tools validate JSON-LD?
Google's Rich Results Test shows what Google can use after rendering, and the Schema Markup Validator checks markup against the full schema.org vocabulary. An AI Crawl Radar audit reads the raw response the way AI crawlers do and reports parse errors and missing recommended types.

See how AI crawlers read your site

Run a free AI visibility audit: a 0–100 score and a fix for every lost point.

Analyze my website

Keep reading

All guides →