CanAIReadMySiteBot
This page is for site owners and sysadmins who see CanAIReadMySiteBot in their logs. It explains what the crawler does, what it sends, and how to block it.
What it does
Can AI Read My Site is a diagnostic tool: a person types a URL into the scanner, and we fetch that one page on their behalf to check whether it's readable by AI crawlers. Every fetch is user-initiated: we do not crawl the web, and a site is only ever requested because a person entered its address.
What a scan requests
A single scan makes at most 3 requests to the same host, in this order:
/robots.txt, always first- the requested page, only if
robots.txtallowsCanAIReadMySiteBoton that page's path /llms.txt, only ifrobots.txtallows it there too
If the page redirects, each redirect target is checked against robots.txt the same way before it is requested: a redirect onto another host reads that host's own robots.txt first. When robots.txt disallows us, the page is simply not requested: the scan says so, reports what robots.txt declares for the AI agents, and labels the result Inconclusive rather than guessing about a page it never read. An unreachable robots.txt (a 5xx or a network error) is treated as a complete disallow, as RFC 9309 requires; a missing one (404) means no restriction.
How it identifies itself
Every request carries the same honest, stable identity, on every fetch, with no exceptions:
User-Agent: CanAIReadMySiteBot/1.0 (+https://canaireadmysite.com/bot) From: hello@canaireadmysite.com
We never send GPTBot, ClaudeBot, PerplexityBot, or any other vendor's user agent. The scanner reports whether your edge blocks self-identifying crawlers as a category, reconciled against what your robots.txt claims, not by pretending to be one of them. We report what we measured (for example, "your server returned HTTP 403 to a self-identifying crawler"), never that a specific named vendor is blocked, unless your robots.txt actually says so. A block is strong evidence; a pass is weaker, since a verified crawler arriving from a published IP range may be treated differently than we are: we don't reproduce IP-range gating, TLS fingerprinting, or reverse-DNS checks (that's the egress IP work below). See the declared-bot access test for the full reconciliation logic, and the crawler reference for the 13 tokens we read from your robots.txt.
Egress IP
Not published yet. Scan traffic currently shares egress with the rest of our infrastructure, so a published IP wouldn't mean much on its own, so we're planning a dedicated, static egress IP with reverse DNS before applying for Verified Bot status.
What it keeps
We do not retain the pages we scan: page text is analysed in memory and discarded when the scan completes, and full page bodies never reach disk. The report itself (the score, each check's status and findings, and the short bot-view extract of your page text, about 500 characters) is kept for two hours so you can export or share it, then permanently deleted by a sweep that runs every 15 minutes whether or not anyone is scanning. A share link shows that stored report for the same two hours and never extends them. A repeat scan of the same page within 30 minutes is answered from a result cache with the same report, and says so.
Separately, a minimal per-page history persists so a rescan of the same page can show what changed: the page address you entered (path and query, never the fragment), its score, which checks passed, and the scoring-model version, but never findings text, page content or the extract. A domain badge record (domain, score, statuses, and the token that lets you embed the badge) is written only when you scan a site's root page. A scan that never observed the page (your robots.txt disallowed us, or nothing answered) is kept for the two-hour export window like any other result but is never added to the history, the badge or the percentile. Rate-limit records keyed by the requester's IP address are kept for 24 hours. The full description is on the About page.
How to block it
A robots.txt rule is the cleanest way:
User-agent: CanAIReadMySiteBot Disallow: /
We honor it, and we check before we fetch: a group naming CanAIReadMySiteBot takes precedence over your User-agent: * group (which we obey when nothing names us), matched by path the way any other crawler would: longest matching rule wins, Allow wins a tie, wildcards and $ supported. A WAF rule matching the exact user agent string above also works if you'd rather block at the edge.
Contact
Questions, abuse reports, or a request to be excluded: contact us or email hello@canaireadmysite.com.
