§ compliance

Do AI crawlers respect robots.txt?

What every major AI operator commits to in writing, the eight documented exceptions - five for user-initiated fetches, three for special-case or integrity crawls - and what independent measurement actually found. Including which crawler has a peer-reviewed finding of non-compliance.

the short answer

Mostly yes, and the popular claim that they routinely ignore it is not supported for the named majors. A six-month controlled study found seven of them obeying robots.txt and one, Bytespider, fetching the file and ignoring it. The real gap is not dishonesty but documented carve-outs: eight crawlers carry a documented exemption, five of them because a human asked for the fetch and three for special-case or integrity crawls.

the full crawler table →

§ in detail

What the documentation actually says.

What the operators commit to

Google is the most absolute: its common crawlers "always obey robots.txt rules when crawling automatically". OpenAI frames the tokens as opt-outs, stating that disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models". Anthropic commits all three of its bots to respecting do-not-crawl signals and additionally honours Crawl-delay. Amazon says its listed agents honour "the user-agent and the allow/disallow directives". Common Crawl documents the opt-out recipe for CCBot. None of that is enforcement: robots.txt is a request that well-behaved clients choose to honour.

The carve-out that matters: a human asked

Eight crawlers carry documented exemptions, and the reasoning is always the same. OpenAI on ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Meta-ExternalFetcher "may bypass robots.txt because it performs fetches that were requested by the user". Google says its user-triggered fetchers "generally ignore robots.txt rules". Amazon's Amzn-User "may not follow all robots.txt directives". Three more are not user-initiated at all: FacebookExternalHit "might bypass robots.txt when performing security or integrity checks", Google's special-case crawlers may ignore it, and Apple's iTMS agent does not follow it.

Anthropic is the outlier, in the direction nobody expects

Every other lab carves out live retrieval. Anthropic documents no user-initiated exception at all, which means Claude-User is held to the same commitment as the training crawler: disallow it and Anthropic documents that its system will not retrieve your content in response to a user query. If you are writing robots.txt on the assumption that "-User" agents ignore you anyway, that assumption is wrong for Claude.

What happens when robots.txt breaks

Only Google documents this properly, and the detail is worth knowing: it caches robots.txt for up to 24 hours (longer if it cannot refresh), and on a 5xx it stops crawling for 12 hours while retrying, then falls back to the last good copy for 30 days, after which an otherwise-healthy site is treated as having no robots.txt at all. All 4xx except 429 are treated as no file. Amazon documents a 30-day cache and fails open. OpenAI, Meta and Perplexity document only a roughly 24-hour propagation delay. Bing documented "a few hours" back in 2012. Anthropic and Apple are silent. So a robots.txt that 500s during a deploy is not a neutral event.

what a 5xx on /robots.txt means to Googlebot

0-12h    stop crawling, keep retrying robots.txt
12h-30d  use the last good copy of robots.txt
30d+     site healthy?  -> behave as if there is no robots.txt
         site also down -> stop crawling

What was actually measured

A peer-reviewed study presented at ACM IMC 2025 (Liu et al., "Somesite I Used To Crawl", arXiv:2411.15091) ran two controlled sites for six months, one with a wildcard disallow and one naming each agent, and watched what arrived. Seven crawlers respected robots.txt: Amazonbot, Applebot, CCBot, ClaudeBot, GPTBot, Meta-ExternalAgent and OAI-SearchBot. Bytespider "fetched the robots.txt file but did not respect it". In active testing, ChatGPT-User and Meta's assistant crawler both honoured robots.txt despite OpenAI's weaker "may not apply" wording. The genuinely alarming finding was elsewhere: of 23 third-party GPT-app crawlers tested, 20 never fetched robots.txt at all.

The Perplexity dispute, with both sides

In August 2025 Cloudflare published a methodologically transparent investigation: brand-new unpublished domains, a blanket-disallow robots.txt, then asking Perplexity about the secret content. Perplexity returned it. Cloudflare reported an undeclared crawler using a generic Chrome user agent, rotating IPs and ASNs across tens of thousands of domains, and de-listed Perplexity as a verified bot. Perplexity responded the same day, saying Cloudflare had confused its traffic with BrowserBase, a third-party cloud browser it uses occasionally, and arguing the principle that user-triggered fetches should follow the user rather than robots.txt. Both positions are public; we are not in a position to adjudicate between them, and neither is anyone else quoting only one of them.

What is not evidence

Percentage claims like "72% of AI crawlers violate robots.txt" trace only to marketing blogs with no retrievable methodology. Litigation is not a finding either: surviving a motion to dismiss means a court assumed the complaint's allegations were true for the purpose of the motion, not that anyone was found to have violated robots.txt. The strongest commitment in the other direction is regulatory: under the EU GPAI Code of Practice (Copyright Chapter, July 2025), signatories commit to employing crawlers that follow RFC 9309. Amazon, Anthropic, Google, IBM, Microsoft, Mistral and OpenAI have signed it. xAI signed only the Safety and Security chapter, and Meta is not listed.

§ who documents what

Who documents what, per operator

The commitment each operator publishes about its own behaviour. A carve-out is not dishonesty: it is a documented policy you can plan around, which is more than an undocumented crawler gives you.

who	verdict	what they say	source
Google (common crawlers)	always, when automatic	"They always obey robots.txt rules when crawling automatically." Special-case crawlers and user-triggered fetchers are documented separately, and both may ignore it.	vendor doc →
Anthropic	yes, no carve-out	All three bots respect do-not-crawl signals, and Crawl-delay is honoured. Uniquely, no exemption is documented for user-initiated fetches, so Claude-User is bound like ClaudeBot.	vendor doc →
OpenAI	yes, except ChatGPT-User	GPTBot and OAI-SearchBot are opt-out tokens. For ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Independent testing nonetheless observed it honouring robots.txt.	vendor doc →
Perplexity	bot yes, user agent no	PerplexityBot is expected to obey. Perplexity-User "generally ignores robots.txt rules" because a user requested the fetch. Contested separately by Cloudflare over undeclared crawling.	vendor doc →
Meta	yes, two exceptions	Meta-ExternalFetcher "may bypass robots.txt" for user-requested fetches, and FacebookExternalHit "might bypass robots.txt when performing security or integrity checks". Changes may take up to 24 hours as robots.txt is cached.	vendor doc →
Amazon	yes, except Amzn-User	Honours user-agent and allow/disallow. Amzn-User "may not follow all robots.txt directives". Caches robots.txt for 30 days and fails open when it cannot fetch the file.	vendor doc →
Apple	yes, with a fallback	Applebot respects robots.txt, and where instructions name Googlebot but not Applebot it follows the Googlebot rules. It does not follow crawl-delay. The separate iTMS agent does not follow robots.txt.	vendor doc →
ByteDance (Bytespider)	measured non-compliant	ByteDance documents nothing. IMC 2025 found Bytespider fetched robots.txt and did not respect it, the only major named crawler with a peer-reviewed finding of non-compliance.	vendor doc →

§ faq

Questions, answered.

Does GPTBot respect robots.txt?

Yes, on both the documentary and the empirical record. OpenAI documents the token as an opt-out from training, and the six-month controlled study presented at ACM IMC 2025 observed GPTBot respecting robots.txt, alongside OAI-SearchBot, ClaudeBot, Applebot, Amazonbot, CCBot and Meta-ExternalAgent. OpenAI has also signed the EU GPAI Code of Practice, which commits signatories to crawlers that follow RFC 9309.

Does ChatGPT-User ignore robots.txt?

OpenAI's wording is that robots.txt rules "may not apply", because the fetch is initiated by a person rather than by automatic crawling. That is weaker than Perplexity's "generally ignores", and in active testing researchers observed ChatGPT-User actually honouring robots.txt. So the honest answer is that OpenAI reserves the right, and the measured behaviour so far has been compliant. Do not rely on either, and decide based on whether you want to be retrievable when someone asks about you.

Which AI crawler is known to ignore robots.txt?

Bytespider, operated by ByteDance, is the only major named crawler with a peer-reviewed finding of non-compliance: the IMC 2025 study found it fetched the robots.txt file and did not respect it. ByteDance publishes no crawler documentation at all. Separately, Cloudflare published a transparent investigation alleging undeclared crawling by Perplexity, which Perplexity disputes. Beyond those two, the circulating claims about major crawlers violating robots.txt do not survive a check of their sources.

How long do robots.txt changes take to take effect?

Roughly a day for most operators, and far longer in one case. OpenAI notes it can take about 24 hours for search results to reflect a change, Meta says crawlers may cache robots.txt for up to 24 hours, and Perplexity says up to 24 hours. Google caches for up to 24 hours and may extend that if it cannot refresh. Amazon is the outlier: it may use a cached copy up to 30 days old. So a robots.txt edit is never an instant switch.

What happens if my robots.txt returns a 500 error?

For Google, a documented sequence: crawling stops for the first 12 hours while it retries, then it uses the last good copy for up to 30 days, and after that a site that is otherwise reachable is crawled as though no robots.txt existed. That last step is the dangerous one, because a long-running 5xx on that single file eventually converts your disallow rules into permission. No other operator documents 5xx handling at all, and Amazon states it behaves as if the file does not exist when it cannot be fetched.

LintPage is not affiliated with any operator or platform named on this page. Vendor claims here were last verified on 2026-09-17. Dashboards and defaults move quickly, so follow the source links before relying on one.

§ related

Next to read.

blockingHow to block AI crawlers.Every documented way to block AI crawlers: robots.txt, Cloudflare, Vercel, Netlify, AWS WAF and Fastly, with the exact settings as the vendors document them today. Including the blocks that cost you more than you think.renderingCan AI crawlers read JavaScript-rendered content?Three crawlers document that they render JavaScript. The AI-native operators document nothing either way. Here is exactly who says what, why the popular answer is an assumption, and how to test your own site in one command.toolRobots.txt CheckerValidate your robots.txt file for syntax errors and blocking rules.

→ robots.txt Testing Guide: How to Test Before You Deploy

§ the measurable part

Find out what they actually get back.

Settings describe intent. A live fetch tells you what your edge really returns to each bot. Free, no signup.

run the AI crawler check →