llms.txt, Explained Honestly

What the file is, what it's supposed to do, a working example, and the part most pages about it skip: no major AI crawler operator documents its crawler reading one. Spec quotes are from llmstxt.org (the proposal, at version 2 as of August 2026), read 13 September 2026.

What it is

llms.txt is a proposed convention for a Markdown file that gives a language model a short, curated map of a site: what it is, and links to the pages worth reading, in a format cheap to parse and cheap to fit in a context window. The proposal describes it as "a proposal to standardise"; it is not a standard any browser vendor, search engine or standards body has ratified. Version 2 of the proposal allows the file at the site root or at any path within it (a /docs/llms.txt covering the pages under /docs/); our scanner checks the site root only.

The actual structure

The proposal is specific about the shape, and only one part of it is required:

An H1 "with the name of the project or site. This is the only required section" (an optional byte-order mark may precede it)
A Markdown blockquote (a line starting with >) "with a short summary of the project, containing key information necessary for understanding the rest of the file"
Zero or more paragraphs or lists of any kind, except headings, with more detail
"Zero or more markdown sections delimited by H2 headers", each a list of links: a required [name](url), then "optionally a : and notes"
By convention, a section headed ## Optional for "secondary information: links an agent can skip when a shorter context is needed"

A working example

# Acme Widgets

> Acme Widgets makes small hardware widgets and publishes API docs, pricing,
> and integration guides for developers building on the Acme platform.

Acme is a developer tool. Most of the useful content for an LLM lives under
/docs. Marketing pages are listed but lower priority.

## Docs

- [Getting started](https://acme.example/docs/getting-started): install the SDK and make your first call
- [API reference](https://acme.example/docs/api): full endpoint list with request/response shapes
- [Authentication](https://acme.example/docs/auth): API keys, OAuth, and token scopes

## Guides

- [Webhooks](https://acme.example/guides/webhooks): receiving and verifying events
- [Rate limits](https://acme.example/guides/rate-limits): quotas and backoff behavior

## Optional

- [Pricing](https://acme.example/pricing): plan tiers and usage-based add-ons
- [Changelog](https://acme.example/changelog): dated list of API changes

This is a minimal example we wrote to illustrate the full recommended structure, not a real site's file.

What our scanner checks

The llms.txt check (5 of 100 points) fetches /llms.txt at the site root and grades it against the full recommended shape, not just the required minimum: an H1, a blockquote summary, and at least one ## section that is actually a link list. All three present is VALID; a file that exists but lacks some of them is WEAK (half credit); no file, or an HTML page in its place, is MISSING; an error status (a 5xx, say) is BROKEN. A file that contradicts itself (some early drafts borrowed Allow:/Disallow: lines from robots.txt, which the proposal itself never defines) is CONFLICTED. Because the proposal requires only the H1, a WEAK file is not an invalid one; it is one that a reader gets less from.

The honest part: what crawlers say about it

The proposal's own site says "thousands of sites publish an llms.txt file", that documentation platforms generate one automatically, and that several AI companies publish one for their own documentation. What nobody has published is a crawler's commitment to read it: none of the operator documentation pages cited on our crawler reference (OpenAI, Anthropic, Perplexity, Google, Apple, Meta) documents its crawler reading the file (some of those documentation sites publish an llms.txt of their own, which is a different thing), while every one of them documents robots.txt, which crawlers built their access model around. That is why our scanner treats llms.txt as a small, cheap-to-earn signal rather than something that gates a page's score. A site without one is not less readable to AI today.

A real example from our own scanning: we checked Anthropic's developer-docs llms.txt (captured from docs.anthropic.com in August 2026; the file now lives at platform.claude.com/llms.txt) against the structure above. It has the H1 and, further down, hundreds of [name](url) entries under ## sections. What it lacks is the recommended blockquote summary (the second line is a plain paragraph), and its first two ## sections (a root URL, a list of available languages) are not link lists. Our scanner grades it WEAK: valid by the proposal's minimum, short of its recommended shape. A major AI company's own file skipping the recommended summary says something about how load-bearing that structure currently is in practice.

Should you still add one?

It's optional. It's cheap, and it earns a small amount of credit in our scoring model (5 of 100 points), and possibly in others'. Our scan lists it as optional guidance, never as something required for AI visibility. Don't expect it to move the needle the way a fast, crawlable, well-structured page does. Those are the things crawlers of every kind depend on regardless of whether this file exists. Our free scan generates a correctly structured llms.txt for your page's detected content, so you don't have to hand-write one to see whether it's worth having.

Run a free scan and get a generated llms.txt →