llms.txt vs robots.txt: One Controls Access, One Is a Reading List

One of these files controls access. The other is a reading list.

robots.txt and llms.txt sit at the same place in your URL structure, are both plain text, and are both aimed at automated clients. That is where the similarity ends, and treating them as two versions of the same idea produces a specific, common, and entirely self-inflicted failure: a site that publishes a curated index for AI assistants while telling those same assistants, one file over, to stay out.

This post covers what each file actually does, the cases people confuse, and the two adjacent comparisons - llms.txt versus llms-full.txt, and the llm.txt spelling that does not exist.

What's in this post

• The one-line difference
• Side by side
• The contradiction
• llms.txt vs llms-full.txt
• llms.txt or llm.txt?
• llms.txt vs schema markup
• Which do you actually need?

The one-line difference

robots.txt says who may fetch what. llms.txt says what is worth fetching.

That is the entire distinction, and almost every practical consequence follows from it. robots.txt is enforcement - voluntary enforcement, but enforcement - with thirty years of crawler behaviour behind it. llms.txt has no enforcement semantics at all. It cannot allow anything, cannot block anything, and has no effect whatsoever on a client that has already been refused access.

Side by side

| | robots.txt | llms.txt | |---|---|---| | Purpose | Access control | Curated index | | Status | De facto standard since 1994, RFC 9309 since 2022 | Proposed, no adopting operator has documented reading it | | Format | Directive lines (User-agent, Disallow, Allow) | Markdown (H1, blockquote, H2 sections of links) | | Obeyed by | Essentially every mainstream crawler | Unknown | | Effect if absent | Everything is crawlable by default | Nothing | | Effect if wrong | Can remove your entire site from search | Nothing measurable today | | Who reads it | Crawlers, before fetching anything else | Clients that already have access |

That last row is the one that matters. robots.txt is read first, and decides whether anything else happens. llms.txt is read by a client that got past that gate.

The asymmetry in the "effect if wrong" row is worth sitting with too. A bad robots.txt is one of the few files on your site that can cost you all of your organic traffic - which is why it gets a validator and a full guide to the ways it goes wrong. A bad llms.txt currently costs you the ten minutes you spent writing it.

The contradiction

Here is the failure this comparison exists to prevent.

# robots.txt
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# llms.txt

> A carefully curated index of our documentation, written for AI assistants.

## Docs

- [Quickstart](https://example.com/docs/quickstart): Get started in ten minutes

Both files are live. Both are valid. Together they say: here is a thoughtfully written index of our site, for the clients we have explicitly forbidden from fetching it.

This happens for mundane reasons. The robots.txt rules were added during the 2023-24 wave of AI opt-outs, possibly by someone else, possibly by a plugin or a platform default. The llms.txt was added later, by someone acting on newer advice, who never looked at robots.txt because why would you - it is a different file about a different thing.

The fix is not automatic, because there is no correct answer that applies to everyone. You have to decide which one you meant:

• You want AI assistants to use your content. Remove the site-wide disallows for the crawlers you want. Keep in mind that training crawlers and live-fetch user agents are different things - GPTBot builds a corpus, ChatGPT-User fetches a page because someone asked about it in a conversation. Many sites want the second and not the first.
• You do not want AI assistants to use your content. Delete the llms.txt. It is doing nothing except signalling the opposite of your actual position.

Whichever you pick, make the two files agree. The llms.txt checker reports this contradiction directly - it reads your robots.txt alongside the file and names which crawlers are blocked - and the AI crawler checker goes further, probing with each bot's real user agent to catch blocks that happen at your CDN and never appear in robots.txt at all.

llms.txt vs llms-full.txt

These two are genuinely related, unlike the pair above.

• llms.txt is the index: links and one-line notes, small enough for a model to read in full before deciding what to fetch.
• llms-full.txt is the expansion: the actual content of those pages, concatenated into one file, so a client can read your documentation without following twenty links.

The practical difference is maintenance. llms.txt is a page you write once and revise occasionally. llms-full.txt is a copy of your content that has to stay in sync with your content - which means it is a build step, not a file you upload. A stale llms-full.txt is worse than not having one: it hands a model a confidently wrong version of your docs.

Rule of thumb: if you cannot generate llms-full.txt as part of your deploy, do not publish one. And if your llms.txt has grown large enough that you are worried about its size, that is the signal to split - the index goes back to being links, the content moves to llms-full.txt.

llms.txt or llm.txt?

llms.txt. Plural, with the s.

llm.txt is a common mistyping and nothing more - there is no competing standard by that name, no client looks for it, and a file served there will not be found by anything. The spec at llmstxt.org is unambiguous, and the domain name itself carries the plural.

If you have already published llm.txt, the fix is to serve the file at llms.txt and either redirect the singular to it or leave it in place; there is no harm in the extra file, and no benefit either.

llms.txt vs schema markup

Occasionally framed as alternatives. They are not comparable.

Structured data describes an entity on a page - this is a product, it costs this much, it has this rating - to a client that is already looking at that page. It is consumed by search engines that have crawled the page and want to render a richer result.

llms.txt operates at the site level and says which pages exist and which are worth reading. It describes nothing about the content of any individual page beyond a one-line note.

They answer different questions and there is no trade-off between them. The difference in maturity is worth noting though: structured data has documented, verifiable effects in Google search results today, and gets them wrong in ways you can measure. llms.txt does not yet have an equivalent.

Which do you actually need?

Everyone needs a robots.txt. Not because it must contain rules - an empty one is a perfectly valid statement that everything is crawlable - but because its absence means a stream of 404s and, on some stacks, a catch-all route answering that path with your app shell. It is also the file that can do real damage, so it is worth validating rather than assuming.

llms.txt is optional and currently unproven. Add one if you want a curated machine-readable index of your site to exist, and because it costs almost nothing if clients do start reading it. Do not add it expecting a measurable change, and do not let its presence substitute for the checks that actually determine whether AI assistants can see your site: whether their crawlers get a 200, and whether your content is there without JavaScript.

llms-full.txt only if you can generate it. Otherwise it will go stale and misrepresent you.

The llms.txt hub covers the format and how to serve it on each platform, including the three where you cannot.