~/search-console/access-forbidden-403

§ page indexing report

blocking

"Blocked due to access forbidden (403)" in Search Console

Your server let Googlebot ask, then refused to answer. Usually a bot-management rule rather than anything you configured on purpose. Here is how to tell.

This URL is not indexed and it should be. Something is in the way.

check a url →

§ check your url

Is it still happening?

Paste the URL. We request it twice from the same place, once with a browser user agent and once with the Googlebot user agent, and show you both status codes side by side.

what this proves

Whether your server responds differently to a crawler than to a browser, which separates a bot-management rule from a genuine access restriction.

what it cannot prove

We request from our own network, not from Google's IP ranges. An edge rule that verifies Googlebot by reverse DNS will treat us as a spoofed crawler, so a 403 here does not always mean the real Googlebot is blocked. URL Inspection in Search Console fetches from Google and settles it.

§ what google is telling you

What this error actually means.

Google requested the page and the server replied 403 Forbidden: request received, understood, and refused. This is not a crawl block and not a missing page. Your infrastructure made a decision about the requester.

Because the response carries no content, Google has nothing to index and nothing to judge. Persistent 403s eventually drop a URL from the index entirely.

The tell that separates the two causes is simple: does the page load for you in a browser? If it does, something is treating crawler traffic differently, and the culprit is almost always a security layer you did not think of as an SEO surface.

§ causes

5 things that produce this error.

Ordered by how often they turn out to be the cause, not by how obvious they look.

A WAF or bot-management rule at the edge

The most common cause by a wide margin. Cloudflare, AWS WAF, Akamai, Sucuri, and most managed hosts ship bot rules that are on by default. They are tuned to block scrapers, and a crawler that is not on the allowlist looks exactly like a scraper. Nothing in your application logs this, because the request never reaches your application.

Rate limiting that a crawl triggers and a human never does

A limit of a few requests per second per IP is invisible to real visitors and routine for a crawler working through a large site. The first few requests succeed and the rest return 403, which is why this error often covers a subset of URLs with no pattern in common.

Authentication in front of content that should be public

Basic auth left on a subdirectory, a preview-protection setting on a hosting platform, a members-only rule that matches more paths than intended, or a paywall that returns 403 instead of serving a public preview.

IP or country blocking

Googlebot crawls predominantly from US IP ranges. A site that geo-blocks traffic outside its own market, or that blocks entire hosting-provider ranges to fight scrapers, blocks the crawler along with everything else.

A user-agent blocklist that is too broad

A rule written to stop one abusive bot, matching on a substring that also appears in legitimate crawler user agents. Blocking on user agent alone is unreliable in both directions: it stops crawlers you want and does nothing against a scraper that sends a browser user agent.

§ the fix

How to clear it.

step 1

Establish whether it is bot-specific

Request the URL twice, once with a browser user agent and once as Googlebot. If the browser gets 200 and the crawler gets 403, you are looking at a bot rule and can stop searching your application code.

bash

curl -sI https://example.com/page -o /dev/null -w "browser: %{http_code}\n"

curl -sI https://example.com/page \
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  -o /dev/null -w "googlebot: %{http_code}\n"

step 2

Allow verified crawlers at the edge, not by user agent

Every major CDN has a verified-bot category that confirms identity by reverse DNS rather than trusting the user-agent string. Use it. An allowlist keyed on the user agent alone is trivially spoofed and gives you no real protection in exchange for the risk of blocking Google.

step 3

Raise or exempt rate limits for verified crawlers

If the block comes from a rate limiter, exempt verified crawler traffic rather than raising the global limit. If crawl volume is genuinely a load problem, set a crawl rate in Search Console instead of returning errors, which is the worst possible signal to send.

step 4

Check the edge logs, not the application logs

A 403 from a WAF never reaches your origin, so your framework logs will show nothing at all for these requests. Look in the CDN or firewall event log, filter for the URL, and read which rule fired. That log entry names the rule, which turns an afternoon of guessing into a one-line change.

§ related

Next to check.

blockingServer error (5xx)Google asked for the page and your server failed to produce it. Often intermittent, often only under crawl load, and the most urgent of the indexing errors.blockingBlocked by robots.txtGoogle was told not to fetch this URL at all. Here is which line of your robots.txt is doing it, why the page can still appear in search anyway, and how to fix it correctly.blockingExcluded by 'noindex' tagGoogle found this page, crawled it, and was told not to index it. Here is where that instruction comes from, the four places it hides, and how to confirm it is gone.toolAI Crawler CheckerTest if ChatGPT, Claude, and Perplexity can actually read your page — not just whether your robots.txt says they can.

→ Why ChatGPT Can't Find Your Site (Even Though robots.txt Says It Can)

LintPage rules covering this: ai/all-crawlers-blocked, security-headers

§ faq

Questions, answered.

Why does the page load fine for me but return 403 to Google?

Because the block is conditional on who is asking. A WAF, bot-management rule, rate limiter, or geo-restriction evaluates the user agent, the IP address, and the request fingerprint. Your browser passes every check; a crawler fails one of them. Your application never sees these requests, so nothing appears in your own logs.

Does Cloudflare block Googlebot?

Not by default, but its bot-management and security rules can, particularly at higher security levels, under "I'm Under Attack" mode, or with a custom rule that matches more traffic than intended. Cloudflare maintains a verified-bot category that identifies real crawlers by reverse DNS. Allowing that category is the correct fix, rather than allowing anything that claims to be Googlebot.

Will a 403 remove my page from Google?

Eventually, yes. Google retries a URL returning 403 for a while, and a page that was already indexed can persist for a time. If the status continues, the URL is dropped. Unlike a 404, a 403 says the resource exists but you may not have it, so Google is somewhat more patient, but not indefinitely.

Should I block AI crawlers the same way?

Only if you have decided to, and only deliberately. The rules that block GPTBot or ClaudeBot are usually the same edge rules that can catch Googlebot, and a broad "block bots" setting rarely distinguishes between them. Decide per crawler, allow verified search crawlers explicitly, and test the result rather than assuming the rule did what its name suggests.

What is the difference between 403 and 401 in Search Console?

401 means authentication is required and was not supplied; 403 means the request was understood and refused regardless of credentials. Google reports them as separate rows. In practice 401 usually points at basic auth or a login wall, while 403 usually points at a firewall, a rate limiter, or an access rule.

§ before the next deploy

Catch this one before Search Console does.

Search Console tells you weeks after the fact. LintPage runs 60 checks against a URL in about 30 seconds. Free, no signup.

run a full scan →