§ blocking

How to block AI crawlers.

Every documented way to block AI crawlers: robots.txt, Cloudflare, Vercel, Netlify, AWS WAF and Fastly, with the exact settings as the vendors document them today. Including the blocks that cost you more than you think.

the short answer

robots.txt is the only control every operator claims to honour, and it is a request rather than a wall. Anything stronger happens at your edge. Both are easy to get wrong in the same direction: blocking the crawler that answers questions about you, while leaving the one that trains on you alone.

the full crawler table →

§ in detail

What the documentation actually says.

Decide which of the three you are blocking

AI traffic splits into training, search indexing, and live retrieval, and they have opposite consequences. Blocking training keeps you out of a future model and costs you nothing today. Blocking a search crawler removes you from that assistant's answers: OpenAI states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Blocking live retrieval means the assistant cannot fetch your page when a person is actively asking about it. Most people who say "block AI" mean the first and accidentally do all three.

robots.txt: the only cross-vendor control

One file, honoured voluntarily. Write one group per token you want to refuse. Note what is NOT in the standard: RFC 9309 defines user-agent, allow and disallow, and nothing else. Crawl-delay is an extension, and support for it is patchy enough that you cannot treat it as a rate limit (see the table below).

/robots.txt

# Opt out of training, stay citable in live answers.

User-agent: GPTBot          # OpenAI training
Disallow: /

User-agent: ClaudeBot       # Anthropic training
Disallow: /

User-agent: Google-Extended # Gemini training and grounding
Disallow: /

# Deliberately absent: ChatGPT-User, Claude-User, Perplexity-User,
# OAI-SearchBot, PerplexityBot. Those are how you get cited.

Cloudflare, where the defaults changed under you

Security Settings > Configure AI bot policies. Search, Agent and Training are set separately, each to Block (on all pages), Block on pages with ads, or Allow. The older "Block AI bots" toggle was deprecated on 15 September 2026, the same day Cloudflare applied new defaults: allow Search, block Training and Agent on pages that display ads. Those defaults were applied to new zones AND to existing free-plan zones whose owners had never changed the setting, so this is worth opening even if you have never configured anything.

Cloudflare, the rest of the surface

AI Crawl Control (the renamed AI Audit) has a Security tab for allowing or blocking individual crawlers, and a Settings tab that picks the block response: 403, or 402 Payment Required if you are charging. Managed robots.txt, under Bot traffic, publishes your preference as content signals and is available on all plans, but it PREPENDS to your existing robots.txt rather than replacing it, so what you serve becomes a concatenation of both. AI Labyrinth is not a block at all: Cloudflare documents that its actions "are not mitigations" and that it neither blocks nor challenges the request.

Vercel, Netlify, AWS and Fastly

Vercel ships an AI Bots managed ruleset that is inactive by default, at Firewall > Rules > Bot Management, where you set it to Log or Deny and publish. (BotID is a different product: an invisible CAPTCHA for checkout and signup routes, not crawler policy.) Netlify blocks through the User Agent Blocker extension, whose own documentation says its list "does not include all possible AI crawlers and AI bots". AWS WAF uses the AWSManagedRulesBotControlRuleSet rule group, where CategoryAI is the one category rule that applies to verified bots as well as unverified ones. Fastly exposes fastly.bot.category.is_ai_crawler and is_ai_fetcher in VCL.

Fastly VCL, from their ContentGuard documentation

if (fastly.bot.category.is_ai_crawler || fastly.bot.category.is_ai_fetcher) {
   error 402 "AI Licensing Required";
}

The blocks that cost more than you meant

Mixed-use crawlers are the trap: Googlebot, Applebot and Bingbot each serve both search and AI, so refusing the AI half refuses search too. Blocking Anthropic by IP can defeat your own opt-out, because it stops them reading the robots.txt where you expressed it. On AWS, turning on Bot Control at Common level blocks verified AI bots via CategoryAI. And if you are in Cloudflare's pay-per-crawl beta, a WAF or Bot Management block overrides the charging feature entirely: you earn nothing instead of being paid.

What a user-agent block cannot do

Any client can send any User-Agent header, so a UA rule filters the polite and nothing else. The verifiable alternatives are IP ranges, which most major operators publish as JSON, and signed requests: OpenAI signs ChatGPT agent traffic using HTTP Message Signatures (RFC 9421) with a Signature-Agent header, and Cloudflare, AWS WAF and Vercel can all verify that signature. If your goal is enforcement rather than preference, that is the layer to work at.

§ who documents what

Who actually honours Crawl-delay

Crawl-delay is not part of RFC 9309. It is an extension, and treating it as a rate limit assumes support that mostly is not documented. Silence is not support.

who	verdict	what they say	source
Anthropic	documented	"We support the non-standard Crawl-delay extension to robots.txt", with an example of Crawl-delay: 1 for ClaudeBot.	vendor doc →
Bing	documented	Honours it, and defines it as a time window of 1 to 30 seconds during which bingbot crawls the site once, not a crawl rate.	vendor doc →
Common Crawl	documented	Honours it for CCBot, and additionally backs off automatically when a server returns 429 or 5xx.	vendor doc →
Google	ignores it	Supports user-agent, allow, disallow and sitemap. "Other fields such as crawl-delay aren't supported."	vendor doc →
Apple	ignores it	"Applebot does not follow crawl-delay."	vendor doc →
OpenAI	silent	Crawl-delay is not mentioned anywhere in OpenAI's crawler documentation. Silence is not a commitment either way, so "gptbot crawl-delay" has no documented answer.	vendor doc →
Perplexity	silent	Crawl-delay is not mentioned anywhere in Perplexity's crawler documentation.	vendor doc →
Meta	silent	Crawl-delay is not mentioned anywhere in Meta's crawler documentation either.	vendor doc →

§ faq

Questions, answered.

How do I block ChatGPT from using my site?

It depends which part you mean, because OpenAI runs three separate crawlers. Disallow GPTBot to stay out of model training. Disallow OAI-SearchBot to leave the ChatGPT search index, which OpenAI says removes you from ChatGPT search answers. ChatGPT-User is the live fetch made when a person asks about your page, and OpenAI documents that robots.txt rules may not apply to it because a user initiated it. Blocking all three is a much bigger decision than blocking the first.

Does blocking GPTBot remove me from ChatGPT answers?

No. GPTBot is the training crawler; blocking it keeps your content out of future model training and changes nothing about what ChatGPT can retrieve or cite today. The crawlers that affect live visibility are OAI-SearchBot, which builds the search index, and ChatGPT-User, which fetches on demand. This is the single most common mix-up in AI crawler advice, and the names are nearly identical, which does not help.

Are AI crawlers blocked by Cloudflare by default?

Partly, and only since 15 September 2026. On that date Cloudflare set defaults for new zones, and for existing free-plan zones whose owners had never changed the setting: Search is allowed, while Training and Agent are blocked on pages that display ads. Before that, from July 2025, new domains were asked at signup rather than defaulted. Because this can have been applied without you choosing it, checking Security Settings > Configure AI bot policies is worthwhile even on a site you never configured.

Does Crawl-delay work for AI crawlers?

Rarely, and it is not part of the robots.txt standard: RFC 9309 defines only user-agent, allow and disallow. Anthropic, Bing and Common Crawl document honouring Crawl-delay. Google and Apple document that they ignore it. OpenAI, Perplexity and Meta do not mention it at all. If you need a real rate limit, apply it at your edge rather than asking politely in a file.

Can I block AI crawlers without losing search traffic?

For the pure-AI crawlers, yes: GPTBot, ClaudeBot and Google-Extended have no bearing on Google Search rankings, and Google states Google-Extended is not a ranking signal. The exception is mixed-use crawlers. Googlebot, Applebot and Bingbot each serve both search and AI, so a block aimed at their AI use takes search with it. That is the trade to understand before you write the rule.

LintPage is not affiliated with any operator or platform named on this page. Vendor claims here were last verified on 2026-09-17. Dashboards and defaults move quickly, so follow the source links before relying on one.

§ related

Next to read.

complianceDo AI crawlers respect robots.txt?What every major AI operator commits to in writing, the eight documented exceptions - five for user-initiated fetches, three for special-case or integrity crawls - and what independent measurement actually found. Including which crawler has a peer-reviewed finding of non-compliance.renderingCan AI crawlers read JavaScript-rendered content?Three crawlers document that they render JavaScript. The AI-native operators document nothing either way. Here is exactly who says what, why the popular answer is an assumption, and how to test your own site in one command.toolAI Crawler CheckerTest if ChatGPT, Claude, and Perplexity can actually read your page — not just whether your robots.txt says they can.

→ Why ChatGPT Can't Find Your Site (Even Though robots.txt Says It Can)

§ the measurable part

Find out what they actually get back.

Settings describe intent. A live fetch tells you what your edge really returns to each bot. Free, no signup.

run the AI crawler check →