Does llms.txt Actually Work? We Measured It

We probed the 10,000 most popular sites, and counted

Most pages about llms.txt still say adoption is growing without saying how they know. The best published measurement is Ahrefs' study of 137,000 sites in Ahrefs Web Analytics, which put adoption at 28% and called that an upper bound, because its customers skew technical.

This post measures a different population - the most popular sites on the web - and checks the part a count cannot: whether what comes back at /llms.txt is an llms.txt at all, and whether it follows the format.

What's in this post

• The headline number
• The false positives
• Adoption by stack
• Quality of the files that exist
• The contradiction
• So does it work?
• Methodology

The headline number

Of 5,889 reachable domains, 1,808 returned a 200 at /llms.txt - 30.7%.

That figure is the one most adoption claims stop at, and it is inflated. Read the next section before quoting it.

The false positives

Of the 1,808 domains returning a 200, 931 returned an HTML document - 51.5% of the apparent adopters.

These are not llms.txt files. They are application shells and error pages. A single-page app, or any framework with a catch-all route, answers an unmatched path with an HTML page and a 200 status, so /llms.txt "exists" in the only sense a status-code check can detect. A model fetching one receives a page of markup and no index at all.

A further 104 returned a 200 with something that is not HTML but is not an llms.txt either - an empty body, a JSON error, a plain-text not-found page, robots.txt rules served at the wrong path. Those are excluded too.

That leaves 773 sites serving an actual llms.txt: 13.1% of the population. A checker that looks only at the status code would report 30.7% - 2.3 times the real figure.

# The check that separates the two. Status code alone cannot.
curl -s https://example.com/llms.txt | head -c 120

Every figure after this point uses the real count, not the 200 count, as its denominator.

Adoption by stack

Detection is best-effort from the homepage response, so anything unrecognised is reported as unknown rather than guessed at. Stacks with fewer than 20 reachable domains are left out of the table entirely: below that, a rate is noise. Two patterns survive those caveats.

The highest rates belong to stacks where adding the file is a settings field or a file in a folder. Webflow now lets a site upload an llms.txt in its SEO settings and serves it from the domain root, which is the likeliest reason its rate sits where it does. Next.js, Astro and Nuxt sites are built by developers, for whom the file is one more thing in a public/ folder.

The second pattern sits below the floor, so it is not in the table: platforms that now write the file for you. Every Shopify store serves a default agent file at /llms.txt unless the theme overrides it, and Wix generates and maintains one for eligible sites. Few of the most popular sites run on either platform, but most of those that do served one, and on Shopify most of those files were the platform's own default - recognisable because Shopify generates the same opening line for every store that has not overridden it. Squarespace has also added a setting where site owners write their own.

That changes what an adoption number means. On a growing share of the web, "has an llms.txt" measures a platform's decision, not a site owner's.

Quality of the files that exist

Of the 773 real files:

• 43.7% break no rule the llmstxt.org spec defines.
• 75.4% carry the > blockquote summary - the single line a model reads to decide whether the rest is worth its context.
• 71.4% annotate at least one link with the : notes suffix that lets a client choose between links without fetching them.
• 15% also serve an /llms-full.txt, the companion file for a client that wants the whole text rather than an index.

Median size is 7,782 bytes across a median of 32 links.

Most of the files that fail do so in the same way: a link section that contains plain bullet points with no link in them. The format defines each H2 section as a list of links, so a bullet that is only prose fails it, even when every link in the file is fine. A smaller group puts something before the H1 or uses several H1s. A few are not markdown indexes at all, but policy statements written as # comment lines, which a markdown reader sees as a stack of headings.

The well-formedness figure counts only spec violations. Our own opinions about these files - that they should be served as text/plain, that past a certain size they have stopped being an index - are excluded from it, because counting our preferences as failures would let us set the bar wherever made the number look most alarming.

The contradiction

64 of the 773 sites serving a real llms.txt - 8.3% - also publish a robots.txt that blocks at least one AI crawler from the entire site.

These sites have written a curated index for clients they have separately forbidden from fetching anything.

It is an easy state to end up in. The robots.txt disallows usually date from the 2023-24 opt-out wave and were often added by a plugin, a platform default, or a colleague. The llms.txt came later, from newer advice, written by someone with no reason to open a different file about a different thing. Nobody made a decision to do both; the site just accumulated them.

The llms.txt vs robots.txt comparison covers which file governs what, and why the resolution has to be a deliberate choice rather than a default.

So does it work?

The data above says who serves the file and how well. Whether anything reads it is a separate question, and it is not observable from the outside: answering it takes either an operator saying so, or server logs from enough sites to show what is fetching the path. On both, the evidence points the same way.

Google says Search does not need it. Its guidance on AI features is explicit that you do not need to create "new machine readable files, AI text files, or markup" to appear in AI Overviews or AI Mode.

No other major operator has documented reading it. OpenAI, Anthropic and Perplexity all publish llms.txt files for their own developer documentation. None of them documents its products fetching yours.

The logs show almost nothing arriving. In Ahrefs' study, 97% of the roughly 38,000 sites publishing the file received no requests for it at all in May 2026. Of the fetches that did happen, the most active named clients were GPTBot and Claude Code, Anthropic's coding agent - and, as the authors point out, a fetch is not the same as a read.

Adoption and use have come apart. 13.1% of the most popular reachable sites serve a real llms.txt, and two of the largest site platforms now add one by default. Almost nothing reads it. A convention becomes worth following when both halves are true, and today only one is.

So: on today's evidence, an llms.txt does not measurably change how AI search sees your site. The file is still cheap - ten minutes and a few kilobytes, no ranking risk - and where a coding agent or a documentation tool does fetch it, a correct file is the version worth having. That makes it a reasonable bet for a documentation site and close to irrelevant for most others. "It will improve your AI visibility this quarter" is not a claim anyone can support.

What does determine whether AI assistants can see your site is duller and entirely measurable: whether their crawlers get a 200 rather than a silent 403 from your CDN, and whether your content exists in the HTML or only after hydration. Those are worth your attention first, and the AI crawler accessibility guide covers what actually goes wrong there.

Methodology

Each domain was fetched directly for this measurement - the homepage for reachability and stack detection, then /llms.txt, /llms-full.txt, and /robots.txt. Files were parsed and validated with the same code that runs the public llms.txt checker, so anything reported here can be reproduced against a single URL.

A response counts as a real llms.txt only if it is not HTML and has an H1 or at least one markdown link. Empty bodies, JSON errors and robots.txt syntax served at the path are excluded; policy files written as # comment lines count, and are reported as malformed. Unreachable domains are excluded rather than counted as non-adopters, because a host that does not respond is not evidence about llms.txt either way. Stack breakdowns are suppressed below 20 domains, where a rate is noise dressed as a finding.

The population's limits are worth stating plainly. Tranco ranks registrable domains by popularity, so this is the top of the web, not the web: some entries are infrastructure domains with no website behind them, and sites behind aggressive bot protection refuse an automated request and drop out. It is one list, measured at one point in time. Repeat runs a few hours apart moved the real-file count by a handful of sites, not by a margin that changes anything above.

The probe identifies itself as LintPageResearch/1.0 in its user agent, with a link to LintPage's llms.txt guides, so anyone who saw it in their logs can find out what it was.

Measured 2026-09-11. The script that produced these numbers is scripts/llms-txt-adoption.ts in the LintPage repository; the figures on this page are read from its output rather than typed into the prose, so they cannot drift from the data.