Methodology

A scan reads robots.txt first, then fetches one page and llms.txt from the same host: at most three requests, static HTML only, no JavaScript executed. The page and llms.txt are only requested where robots.txt allows CanAIReadMySiteBot; if it doesn't, the result is labelled Inconclusive and says which rule applied, rather than scoring a page we never read. The same label applies when the page gives no response at all (a connection error or timeout): one attempt can't tell a site that is down from a transient problem on our side, so that is reported as uncertainty, never as a block. Results are per page (the exact address you enter, path and query included), and a repeat of the same page within 30 minutes is served from our result cache and says so. Below is every check that goes into the result: what it measures, why it matters, and how it feeds the score. See /bot for exactly what the scanner sends.

Three kinds of evidence

Every check in a result is labelled with the kind of evidence it rests on, because the three are not interchangeable:

Crawl access

Can a crawler reach and parse this page at all? This covers whether the page loads with a normal HTTP status, whether robots.txt permits the major AI agent tokens (see the full reference) for the page you scanned, evaluated at that page's own path and query string, the way each crawler would, with every group naming an agent combined and the * group used only when nothing names it. We also check whether the page has meaningful text without running JavaScript: a page that's an empty shell until a client-side framework renders it is invisible to a crawler that only reads HTML. This matters most: nothing else on this page can be checked if a crawler can't get in the door.

The declared-bot access test

We identify honestly, every time, as CanAIReadMySiteBot: we never send GPTBot, ClaudeBot, PerplexityBot, or any other vendor's user agent. See /bot for exactly what that means and why impersonating a crawler was never on the table. That one honest fetch is reconciled against what your robots.txt says about CanAIReadMySiteBot for that page (a group naming us, or your * group when none does):

We report what was measured ("your server returned HTTP 403 to a self-identifying crawler"), never "GPTBot is blocked," unless your robots.txt actually says so. A block is strong evidence. A pass is weaker: a verified crawler arriving from a published IP range may be treated differently than we are, and we don't reproduce IP-range gating, TLS fingerprinting, or reverse-DNS checks.

llms.txt

llms.txt is optional guidance, and the scan treats it that way. We check whether the file exists, whether it parses, whether it follows the proposal's recommended structure (an H1, the only part the proposal requires, plus a blockquote summary and ## sections that are actual link lists), and whether it contradicts itself: a file that both allows and disallows the same path is flagged, not silently accepted. This category is deliberately weighted lightly: no crawler operator documents its crawler reading the file, and a site without one is not less readable to AI. It earns points; it is not required for AI visibility and never gates the verdict. See the honest explainer for what we mean by that.

Training crawlers

Retrieval agents and training crawlers are scored separately. The training check looks only at whether your policy is internally consistent: blocking every training crawler, allowing every one, or any deliberate mix all earn full marks. The single point it carries is lost only for a same-operator gap: ClaudeBot disallowed while anthropic-ai, a token many robots.txt files name for Anthropic although its documentation does not mention it (it lists only ClaudeBot, Claude-User and Claude-SearchBot), is left allowed. That usually means one token was missed, not that a choice was made. When it fires, the fix we list is to extend the opt-out, never to remove it.

Content readability

• Whether the page has a substantial amount of visible text once scripts, styles, and markup are stripped out; a JavaScript-heavy page doesn't score better for shipping more JavaScript; only the actual visible content counts
• Heading presence and hierarchy: a page with no H1, or one buried under a mess of competing headings, is harder for anything (human or machine) to summarize correctly

Structure and entity clarity

Whether the page clearly identifies what it's about at a structural level: organization, product, or person, and whether structured data (schema.org) is present to make that explicit rather than merely implied by prose.

Intent clarity

Whether the page's <title> tag agrees with its H1 and visible content. A page whose title says one thing and whose heading says another sends a mixed signal about what it's actually for, to a crawler and to a human skimming a search result alike.

Trust and publisher signals

Whether basic accountability signals are present in the HTML: an About or Contact link, a footer, and (for content-heavy pages like articles) an author meta tag. These are cheap to add. We check for their presence only; we do not measure how any search engine or AI system weighs them.

How it becomes one number

The model is published in full so a score can be checked by hand from the result card. Every check ends in one status; the status tier decides how much of the check's weight it earns; the weights sum to 100; then a small set of hard ceilings applies; then the number becomes a label. Nothing else feeds the score.

Status tiers

Weights

The result card groups the first four as Crawl 30, then Content 30 · Entity 15 · Intent 10 · Trust 10 · Guidance 5. The declared-bot access test above is reported alongside these but carries no weight: crawl.access, the two robots.txt checks and the ceilings below already count the observations it reconciles, and scoring it too would count them twice.

Unknown states

A check that could not run reports UNKNOWN rather than a guess: robots.txt or llms.txt that was unreachable, refused (401/403) or too large to read safely, a page body over our size limit, a rate-limit response from your server, or a 403 that your robots.txt does not corroborate. An UNKNOWN check is removed from both the points earned and the points possible, and the rest is rescaled to 100; it is never credited as a pass and never charged as a failure. The arithmetic line on the result card shows the reduced total, so a score with an excluded check is visibly out of fewer points. When the page itself was never observed, the checks that read its body are all UNKNOWN and the label is Inconclusive (below).

Ceilings

After the weighted sum, a small set of hard ceilings applies. When more than one holds, the lowest wins. A ceiling can only lower a score, never raise it.

84 is one below the AI-Readable line on purpose: a page that blocks even one retrieval agent cannot carry that label, however polished the rest is. There is deliberately no ceiling for training policy: blocking training crawlers is a rights decision, and the model never treats it as a blocker, quietly or otherwise.

From number to label

The split below 50 exists because the raw score conflates three questions (can a crawler reach and parse this page, how much content is there, and how many entity and trust signals does it carry), and reporting a low score on the third as if it were the first would tell the modal small-business or personal site something false about a page we read just fine. The label says which situation is true; the number underneath still says how much signal is there.

Scoring version

The model above is scoring version 2026-09-15. Every result carries the version it was measured under. A rescan is compared only with a previous measurement of the same page made under the same version, and the percentile line counts only pages measured under the current version, so a change to the model can never show up as your site improving or regressing. The model is pinned against 17 real pages captured offline (homepages, docs, articles and a product page, from anthropic.com and nytimes.com to danluu.com and example.com), and their expected scores are part of the test suite; any change that moves one of them changes the version, and this page is updated in the same change.

What the free report includes beyond the score

The diagnosis comes with the fixes that apply, not just a verdict: every free scan lists one fix per check that fell short, derived only from the results, with robots.txt items stated as what the file declares and content items as heuristic suggestions. It also generates a ready-to-upload llms.txt for the page's detected content plus a robots.txt block reflecting the AI-agent policy we found, both downloadable directly from the result. Each "fixing X → N" line is the score this model would report if that check reached its best status; it is not a prediction of traffic, citations or ranking.

Limitations: what a scanner can't tell you

• Static HTML only. No JavaScript is executed, so client-rendered content is invisible to the scan, which is the point, since that's also what a crawler that doesn't render JS would see, but it means the score reflects the raw HTML delivered, not whatever a browser would eventually paint.
• One page, one scan. Results describe the single URL you gave us, not your whole site.
• We never impersonate a crawler. The edge test measures whether declared, self-identifying crawlers are blocked as a category, not whether one specific named vendor is singled out. Per-agent detail comes entirely from parsing your robots.txt, never from sending a vendor's user agent.
• A block is strong evidence; a pass is weaker. Sites can also gate on IP range, TLS fingerprint, or reverse DNS, none of which we reproduce or can observe from outside. A verified vendor crawler arriving from its published range may be let in where we were refused, or refused where we were let in.
• We do not observe assistants. Nothing in a result measures whether ChatGPT, Claude, Perplexity or any other assistant reads, understands, mentions, cites or ranks a page, or sends it traffic. The score is a diagnostic of reachability, declared policy and structure.
• Point-in-time. A CDN setting, a WAF rule, or a plugin update can change the answer tomorrow. Results are a snapshot, not a guarantee.
• Single geography. All scans originate from one location. A site that serves different content or applies different blocking rules by region will look different from where you're sitting than it does to us.

If a result looks wrong for a reason this page doesn't cover, tell us: the pinned reference pages exist so that a disputed score can be traced to a specific check.

Run a free scan →

See also: the AI crawler reference, llms.txt explained, how our own bot identifies itself.