A markdown file at your site root that nobody has proven anyone reads
llms.txt is a proposed standard for a plain markdown file at https://yoursite.com/llms.txt that hands AI assistants a short, curated index of your site. It was proposed by Jeremy Howard in September 2024, the spec lives at llmstxt.org, and the format is small enough to learn in about ninety seconds.
That is the easy part. The harder part, and the reason most write-ups on this topic are useless, is that the file's actual effect is unproven. Pages explaining llms.txt tend to describe the format, imply it improves your visibility in ChatGPT, and let the implication do the work. This one separates the two: here is what the file is, and here, separately, is what is known about whether it does anything.
What's in this post
- The format, in full
- What each part is for
- What llms.txt is not
- Does it do anything yet?
- How to write one worth having
- The failure nobody checks for
The format, in full
This is not an abridged version. The entire format is an H1, an optional blockquote, optional prose, and H2 sections containing lists of links.
# Project name
> One sentence describing what this is.
Optional free-form prose. No headings in here.
## Docs
- [Quickstart](https://example.com/docs/quickstart): Auth and a first request in ten minutes
- [API reference](https://example.com/api): Every endpoint, with request and response shapes
## Optional
- [Changelog](https://example.com/changelog): Release history back to v1
Exactly one element is required: the H1. Everything else the spec describes as the expected shape rather than a hard rule. In practice you want all of it, because a file with no summary and no link notes is a list of URLs, and your sitemap already does that better.
What each part is for
The H1 is the project name
Not a page title, not a tagline. The name of the thing whose documentation this indexes. A client reads the first H1 as the project name, so if you have two, the first one wins - which may not be the one you meant.
The blockquote is the most valuable line in the file
> One sentence describing what this is. This is what a model reads to decide whether the rest of the file is worth its context budget. It is technically optional and you should always write one. Make it say what the thing does and who it is for, not what your marketing site says.
H2 sections group the links
Name them for what they contain: Docs, API, Guides, Examples. There is no controlled vocabulary, with one exception below.
## Optional is a reserved section name
This is the piece most summaries of the format leave out, and it is the most interesting design decision in the spec. A section literally named Optional marks content a client may skip when it is short on context. It is an explicit priority signal - you telling a model what to drop first. Put your changelog, your about page, and your legal pages there.
The : notes suffix is the whole point
- [Webhooks](https://example.com/docs/webhooks): Event types, retry behaviour, and signature verification
The note exists so a model can choose between links without fetching them. This is where a hand-written file beats a generated one every time: a note lifted from your meta description is written to earn a click from a human in a search result, and usually says nothing about what the page contains.
What llms.txt is not
Three confusions account for most of the bad advice on this topic.
It is not robots.txt. robots.txt is an access-control convention crawlers have obeyed for thirty years - it says who may fetch what. llms.txt grants nothing and blocks nothing. If your robots.txt disallows GPTBot, publishing an llms.txt does not invite it in; the block is what it obeys. That contradiction is common enough that our checker reports it, and the llms.txt vs robots.txt comparison covers where each one actually applies.
It is not a sitemap. A sitemap is exhaustive, machine-readable, and written for a crawler that will fetch everything anyway. An llms.txt is short, curated, and annotated in prose, written for a client with a limited context budget that has to choose. Generating one from the other strips out exactly the part that makes it useful.
It is not schema markup. Structured data describes an entity on a page to a search engine that is already on that page. llms.txt is a site-level index that tells a client which pages exist. They do not overlap and neither substitutes for the other.
Does it do anything yet?
Here is the honest state of it, as of writing.
No major AI assistant operator has documented reading llms.txt to decide what to retrieve or cite. Not OpenAI, not Anthropic, not Perplexity, not Google. That is not the same as proof that none of them do - absence of documentation is weak evidence - but it does mean every confident claim you have read about llms.txt improving AI visibility is an inference, not a finding.
The case studies claiming a lift almost universally share a shape: a site added llms.txt, other things changed at the same time, AI referral traffic rose, and the post attributes the rise to the file. No mechanism is proposed, no control is described, and the traffic measurement itself is usually a referrer header that AI assistants send inconsistently.
Meanwhile the argument for adding one does not depend on any of that. The file takes ten minutes to write. It is a few kilobytes. It cannot hurt your rankings, cannot block anything, and if clients do start reading it you will already be there. That is a perfectly good reason to add one. "It will improve your AI visibility this quarter" is not.
If you want to know what actually determines whether an AI assistant can see your site today, it is far more boring: whether its crawler can fetch your pages at all, whether your content survives without JavaScript, and whether a CDN rule is quietly returning 403 to it. Those are measurable right now, and they are what the AI crawler accessibility guide is about.
How to write one worth having
Assume a model gets your llms.txt and nothing else, and has to decide what to fetch next.
- Name the project and say what it is. H1, then one blockquote sentence. If a reader cannot tell what you do from those two lines, nothing below them will help.
- Include the pages you would send a new engineer to. Not every page. Not your sitemap. The ten to thirty URLs that actually explain the thing.
- Write a note for every link, describing the destination. "Event types, retry behaviour, and signature verification" - not "Learn more about our powerful webhooks."
- Put skippable content under
## Optional. Changelog, about, legal, press. - Keep it small. If it has grown past a few dozen kilobytes it has stopped being an index. That content belongs in
llms-full.txt, which exists precisely sollms.txtcan stay short. - Serve it as text.
text/plainortext/markdown, at the domain root.
The failure nobody checks for
The most common way a correct llms.txt fails has nothing to do with the format.
If your site is a single-page app, or any framework with a catch-all route, a request for /llms.txt that does not match a static file falls through to your application and gets answered with your HTML shell - and a 200 status. Every tool that checks whether the file "exists" by looking at the status code reports success. A model fetching it receives a page of markup and no index.
# The only check that matters. If this prints <!DOCTYPE html>, your
# llms.txt is not being served, whatever the status code says.
curl -s https://example.com/llms.txt | head -c 120
The same class of failure has a different cause on every stack: middleware intercepting the path before static files are reached on Next.js, a 404 template rendered with a 200 status on WordPress, the domain root not being writable at all on Shopify, Squarespace, and Webflow. The platform guides cover each one, and the llms.txt checker inspects the response body rather than trusting the status code - and tells you which of its rules come from the spec and which are our opinion.