llms.txt vs robots.txt: One Controls Access, One Is a Reading List

One of these files controls access. The other is a reading list.

robots.txt and llms.txt sit at the same place in your URL structure, are both plain text, and are both aimed at automated clients. That is where the similarity ends, and treating them as two versions of the same idea produces a specific, common, and entirely self-inflicted failure: a site that publishes a curated index for AI assistants while telling those same assistants, one file over, to stay out.

This post covers what each file actually does, the cases people confuse, and the comparisons that keep coming up around it.

What's in this post

• The one-line difference
• Side by side
• The contradiction
• llms.txt vs llms-full.txt
• llms.txt or llm.txt?
• llms.txt vs Agent Skills
• llms.txt vs schema markup
• Which do you actually need?

The one-line difference

robots.txt says who may fetch what. llms.txt says what is worth fetching.

That is the entire distinction, and almost every practical consequence follows from it. robots.txt is enforcement - voluntary enforcement, but enforcement - with thirty years of crawler behaviour behind it. llms.txt has no enforcement semantics at all. It cannot allow anything, cannot block anything, and has no effect whatsoever on a client that has already been refused access.

Side by side

That last row is the one that matters. robots.txt is read first, and decides whether anything else happens. llms.txt is read by a client that got past that gate.

The asymmetry in the "effect if wrong" row is worth sitting with too. A bad robots.txt is one of the few files on your site that can cost you all of your organic traffic - which is why it gets a validator and a full guide to the ways it goes wrong. A bad llms.txt currently costs you the ten minutes you spent writing it. How many sites have spent those ten minutes, and what the resulting files look like, are measured here.

The contradiction

Here is the failure this comparison exists to prevent.

# robots.txt
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# llms.txt

> A carefully curated index of our documentation, written for AI assistants.

## Docs

- [Quickstart](https://example.com/docs/quickstart): Get started in ten minutes

Both files are live. Both are valid. Together they say: here is a thoughtfully written index of our site, for the clients we have explicitly forbidden from fetching it.

This happens for mundane reasons. The robots.txt rules were added during the 2023-24 wave of AI opt-outs, possibly by someone else, possibly by a plugin or a platform default. The llms.txt was added later, by someone acting on newer advice, who never looked at robots.txt because why would you - it is a different file about a different thing.

The fix is not automatic, because there is no correct answer that applies to everyone. You have to decide which one you meant:

• You want AI assistants to use your content. Remove the site-wide disallows for the crawlers you want. Keep in mind that training crawlers and live-fetch user agents are different things - GPTBot builds a corpus, ChatGPT-User fetches a page because someone asked about it in a conversation. Many sites want the second and not the first.
• You do not want AI assistants to use your content. Delete the llms.txt. It is doing nothing except signalling the opposite of your actual position.

Whichever you pick, make the two files agree. The llms.txt checker reports this contradiction directly - it reads your robots.txt alongside the file and names which crawlers are blocked - and the AI crawler checker goes further, probing with each bot's real user agent to catch blocks that happen at your CDN and never appear in robots.txt at all.

llms.txt vs llms-full.txt

These two are genuinely related, unlike the pair above.

• llms.txt is the index: links and one-line notes, small enough for a model to read in full before deciding what to fetch.
• llms-full.txt is the expansion: the actual content of those pages, concatenated into one file, so a client can read your documentation without following twenty links.

The practical difference is maintenance. llms.txt is a page you write once and revise occasionally. llms-full.txt is a copy of your content that has to stay in sync with your content - which means it is a build step, not a file you upload. A stale llms-full.txt is worse than not having one: it hands a model a confidently wrong version of your docs.

Rule of thumb: if you cannot generate llms-full.txt as part of your deploy, do not publish one. And if your llms.txt has grown large enough that you are worried about its size, that is the signal to split - the index goes back to being links, the content moves to llms-full.txt.

llms.txt or llm.txt?

llms.txt. Plural, with the s.

llm.txt is a common mistyping and nothing more - there is no competing standard by that name, no client looks for it, and a file served there will not be found by anything. The spec at llmstxt.org is unambiguous, and the domain name itself carries the plural.

If you have already published llm.txt, the fix is to serve the file at llms.txt and either redirect the singular to it or leave it in place; there is no harm in the extra file, and no benefit either.

llms.txt vs Agent Skills

The newest version of this confusion, and the one where the two things are furthest apart.

An Agent Skill, as Anthropic documents it, is a folder containing a SKILL.md file: YAML frontmatter carrying a name and a description, a body of instructions, and optionally bundled reference files and scripts. The loading model is specified rather than assumed. The frontmatter sits in the system prompt from startup, the body is read only when a request matches the description, and bundled files are opened only if those instructions call for them.

Read that and the resemblance is obvious. Both are markdown, both put a short description in front of the detail, and both exist so that a model can decide what is worth reading before reading it. The similarity is real, and it is also where the comparison ends.

llms.txt is something you publish. A Skill is something someone installs. Your llms.txt sits at your domain root and is fetched, in theory, by a client that came to your site. A Skill lives on the filesystem of the agent using it: ~/.claude/skills/ or .claude/skills/ in Claude Code, a zip uploaded through claude.ai settings, or the /v1/skills endpoint on the API. There is no URL on your own domain where you can put a Skill and have visiting assistants pick it up. Nothing in Anthropic's Skills documentation describes one.

That settles the question people are actually asking, which is usually whether to write a Skill instead of an llms.txt. You cannot. A Skill you write is installed by you, into your own agent, for your own work. It never reaches an assistant that somebody else is running against your site. If your goal is for your documentation to be usable by an assistant you do not control, llms.txt is at least aimed at that problem, even if nothing has been documented to read it.

Three asymmetries are worth keeping straight:

• Only one side's reading is documented. Anthropic specifies when a Skill is loaded and what it costs in context. Nobody specifies what reads llms.txt, which is the entire finding of the measurement across the top 10,000 sites.
• Only one side is code. Anthropic's guidance is to use Skills only from trusted sources, because a Skill can bundle scripts that the agent executes, and a malicious one can direct that agent to misuse its tools. An llms.txt is inert text. Whatever the two formats share in shape, their risk profiles have nothing in common.
• Only one side can reach your site at all. Skills used through the Claude API run in a sandboxed container with no network access. A Skill running there cannot fetch your pages, whatever your llms.txt says.

llms.txt vs schema markup

Occasionally framed as alternatives. They are not comparable.

Structured data describes an entity on a page - this is a product, it costs this much, it has this rating - to a client that is already looking at that page. It is consumed by search engines that have crawled the page and want to render a richer result.

llms.txt operates at the site level and says which pages exist and which are worth reading. It describes nothing about the content of any individual page beyond a one-line note.

They answer different questions and there is no trade-off between them. The difference in maturity is worth noting though: structured data has documented, verifiable effects in Google search results today, and gets them wrong in ways you can measure. llms.txt does not yet have an equivalent.

Which do you actually need?

Everyone needs a robots.txt. Not because it must contain rules - an empty one is a perfectly valid statement that everything is crawlable - but because its absence means a stream of 404s and, on some stacks, a catch-all route answering that path with your app shell. It is also the file that can do real damage, so it is worth validating rather than assuming.

llms.txt is optional and currently unproven. Add one if you want a curated machine-readable index of your site to exist, and because it costs almost nothing if clients do start reading it. Do not add it expecting a measurable change, and do not let its presence substitute for the checks that actually determine whether AI assistants can see your site: whether their crawlers get a 200, and whether your content is there without JavaScript.

llms-full.txt only if you can generate it. Otherwise it will go stale and misrepresent you.

Agent Skills answer a different question entirely. Write one to make your own agent work your way. It is not a channel to anybody else's, and it is not something you can publish for visiting assistants to find.

The llms.txt hub covers the format and how to serve it on each platform, including the three where you cannot.