About Can AI Read My Site?
A read-only diagnostic for one question about one page: can the crawlers behind AI assistants reach it, are they allowed to, and does the HTML they receive make sense without a browser. You paste an address; we fetch that page once, as ourselves, and show you what we got. It is a utility, not a suite, and it makes no promises about rankings, traffic or mentions.
What it measures, and what it doesn't
Every check in a result is labelled with the kind of evidence it rests on: observed (what our own self-identifying request received from the page, and whether /llms.txt answered), declared (what the site's robots.txt states for the 13 AI agent tokens the model scores: the ones their operators document, plus the legacy anthropic-ai token real files still name), or heuristic (a structural reading of the HTML: visible text, headings, structured data, title/description agreement, publisher links). What we observe is how our own crawler was treated: a block is strong evidence, a pass is weaker, because a verified vendor crawler arriving from its published address range may be treated differently. We do not observe any assistant, and nothing here measures whether ChatGPT, Claude, Perplexity or anything else reads, cites or ranks a page. The report includes every finding, untruncated, and every fix that applies. The full scoring model (weights, ceilings, thresholds and version) is on the methodology page.
How a scan works
A scan reads your robots.txt first, then fetches the one page you named, then /llms.txt: at most three requests to your host, each carrying the same honest identity: User-Agent: CanAIReadMySiteBot/1.0 (+https://canaireadmysite.com/bot) and From: hello@canaireadmysite.com. We never send another vendor's user agent. If your robots.txt disallows CanAIReadMySiteBot for that page, the page is not requested and the result says so; every redirect hop is checked the same way. No JavaScript is executed: the page is parsed as the static HTML our request received; a crawler that renders JavaScript may see more. Addresses that resolve to private networks are refused outright. A scan that times out or fails inside our system is an error, never a score. Results are per page, point-in-time, and made from a single server location; details are on the crawler page and under limitations.
How it is built and checked
One small Python service: FastAPI for the site, lxml for HTML parsing, WeasyPrint for the PDFs. There is no database server (rate limits, the per-page history and the badge records live in a single SQLite file) and no client-side framework. It runs as one container, as a non-root user with memory and CPU limits, behind HTTPS. The scoring model is pinned against 17 real pages captured offline, from anthropic.com and nytimes.com to danluu.com and example.com; their expected scores are part of the automated test suite, and any change that moves one of them changes the published scoring version. The crawler tokens in the reference are read from each operator's own documentation, and the page carries the date those sources were last read.
What we keep
We do not retain the pages we scan: page text is analysed in memory and discarded when the scan completes, and full page bodies never reach disk. The report itself (the score, each check's status and findings, and the short bot-view extract of your page text, about 500 characters) is kept for two hours so you can export or share it, then permanently deleted by a sweep that runs every 15 minutes whether or not anyone is scanning. A share link shows that stored report for the same two hours and never extends them. A repeat scan of the same page within 30 minutes is answered from a result cache with the same report, and says so.
Separately, a minimal per-page history persists so a rescan of the same page can show what changed: the page address you entered (path and query, never the fragment), its score, which checks passed, and the scoring-model version, but never findings text, page content or the extract. A domain badge record (domain, score, statuses, and the token that lets you embed the badge) is written only when you scan a site's root page. A scan that never observed the page (your robots.txt disallowed us, or nothing answered) is kept for the two-hour export window like any other result but is never added to the history, the badge or the percentile.
Beyond that: rate-limit records keyed by your IP address (scan timestamps and the per-site cooldown), kept for 24 hours. Pages on this site load two analytics scripts, Google Analytics and Simple Analytics; neither is gated on a consent prompt or the Do Not Track signal. The PDF reports load nothing.
Who to contact
Questions about a result, a correction to a reference page, a request to be excluded from scanning, or an abuse report: hello@canaireadmysite.com (also on the contact page). That address is the one in the crawler's From header, so a sysadmin reading it in a log reaches the same place.
Supporting it
The scanner is free to use. If it saved you time, you can buy me a coffee; it helps keep the scanner running.
