~/search-console/blocked-by-robots-txt

§ page indexing report

blocking

"Blocked by robots.txt" in Search Console

Google was told not to fetch this URL at all. Here is which line of your robots.txt is doing it, why the page can still appear in search anyway, and how to fix it correctly.

This URL is not indexed and it should be. Something is in the way.

check a url →

§ check your url

Is it still happening?

Paste the blocked URL. We fetch the live robots.txt for that host, work out which group applies to Googlebot, and apply the longest-match rule the way a crawler does.

what this proves

Whether Googlebot is allowed to fetch this exact path, and which line of your robots.txt decides it.

what it cannot prove

We evaluate the file as Googlebot would. Other crawlers match different groups and may reach a different answer, and a file that was only just deployed may not be the one Google has cached.

§ what google is telling you

What this error actually means.

Google never fetched this page. It read your robots.txt, found a rule covering this path, and stopped before making the request. It does not know what is on the page, whether it has a title, or whether it is any good.

The important consequence: because Google never saw the page, it never saw any instruction on the page either. A noindex tag on a robots-blocked URL is unreachable and does nothing.

A blocked URL can still show up in search results. If other pages link to it, Google can index the URL itself on the strength of those links, listing it with no description and a note that no information is available. Blocking is not removal.

§ causes

5 things that produce this error.

Ordered by how often they turn out to be the cause, not by how obvious they look.

A blanket disallow that shipped from staging

The two-line robots.txt that keeps a staging environment out of the index, deployed to production along with everything else. It blocks the entire site, and nothing on the site looks any different.

text

User-agent: *
Disallow: /

A rule that is broader than it looks

robots.txt paths are prefix matches, not exact ones. Disallow: /blog blocks /blog, /blog/post, and also /blogging-tips. Disallow: /api blocks anything starting with those four characters. A missing trailing slash quietly widens the blast radius.

text

Disallow: /blog        # blocks /blogging-tips too
Disallow: /blog/       # blocks only what is under /blog/

Group precedence that does not work the way it reads

A crawler obeys exactly one group: the most specific one matching its name. If there is a User-agent: Googlebot group anywhere in the file, Googlebot uses that group and ignores the User-agent: * group entirely, including any Allow rules in it. Adding a Googlebot group to grant an exception routinely removes permissions instead.

text

User-agent: *
Allow: /

User-agent: Googlebot
Disallow: /products/    # Googlebot obeys ONLY this group

Blocking the assets the page needs to render

Disallowing /assets/, /static/, or /_next/ blocks the CSS and JavaScript Google needs to render the page. The HTML is fetched, the render is broken, and the page is judged on whatever survived. This does not produce this error on the page itself, but it is why it is worth reading a robots.txt in full rather than fixing one line.

A robots.txt that is not being served as one

A robots.txt that returns 404 is treated as "crawl everything", which is usually fine. One that returns 500 is treated as "crawl nothing" until it recovers. A file served with the wrong content type, or an SPA that returns index.html for every path including /robots.txt, produces rules nobody wrote.

§ the fix

How to clear it.

step 1

Identify the exact rule before editing

Fetch the live file rather than reading the one in your repo. They diverge more often than you would expect, especially when a CDN, a framework route, and a static file all claim the same path.

bash

curl -s https://example.com/robots.txt

step 2

Narrow the rule instead of deleting the file

Replace a broad Disallow with specific paths. Keep blocking what genuinely wastes crawl budget: faceted filter URLs, internal search results, session parameters, and admin routes. Leave everything you want ranking reachable.

text

User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /*?sort=
Allow: /

Sitemap: https://example.com/sitemap.xml

step 3

Use noindex, not robots.txt, to remove a page from search

These solve different problems and are not interchangeable. To keep a page out of the index, allow Google to crawl it and serve a noindex directive. To stop a crawler wasting requests on URLs that do not matter, use robots.txt. Using robots.txt for removal leaves URLs indexed with no snippet and no way for Google to learn otherwise.

step 4

Test the specific URL, then validate the fix

Check the exact path Search Console flagged, not the homepage. Then use Validate Fix in the Page indexing report. Note that robots.txt itself is cached by Google for around 24 hours, so a corrected file is not necessarily in effect the moment you deploy it.

§ related

Next to check.

blockingExcluded by 'noindex' tagGoogle found this page, crawled it, and was told not to index it. Here is where that instruction comes from, the four places it hides, and how to confirm it is gone.blockingURL is unknown to GoogleGoogle has never seen this URL. Nothing is broken and nothing is blocked; there is simply no route to it. Here is how to build one.blockingBlocked due to access forbidden (403)Your server let Googlebot ask, then refused to answer. Usually a bot-management rule rather than anything you configured on purpose. Here is how to tell.toolRobots.txt ValidatorValidate your robots.txt file for syntax errors and blocking rules.

→ robots.txt Testing Guide: How to Test Before You Deploy

LintPage rules covering this: robots-txt-check, sitemap-presence

§ faq

Questions, answered.

Can a page blocked by robots.txt still appear in Google?

Yes. Google can index a URL it has never fetched if enough other pages link to it. The result appears with the bare URL and a line saying no information is available. This is why robots.txt is the wrong tool for keeping something out of search results, and why a page you blocked for privacy reasons can still be listed by name.

Should I use robots.txt or noindex?

Use robots.txt to stop crawlers wasting requests on URLs that will never rank: internal search, faceted filters, infinite parameter spaces. Use noindex to keep a page out of search results. If you need a page removed from the index, it must remain crawlable so Google can see the noindex. Applying both at once is self-defeating, because the block prevents the directive from ever being read.

How long does Google cache robots.txt?

Roughly 24 hours, though it varies with how often your site is crawled and Google may refetch sooner if the request fails. A newly corrected robots.txt is therefore not always in effect immediately. You can prompt a refetch through the robots.txt report in Search Console.

Does the order of rules in robots.txt matter?

Not the order, but the specificity does. Within the one group that applies to a crawler, the longest matching path pattern wins, and when an Allow and a Disallow match with equal length the Allow wins. What does matter structurally is which group applies at all: a crawler obeys only the most specific group matching its name and ignores every other group in the file.

Why does my robots.txt look correct in the repo but not live?

Because something else is serving that path. A CDN rule, an edge function, a framework route handler, or a catch-all that returns index.html for unmatched paths can all shadow a static file. Always check the response from the live URL rather than the file in version control, and confirm it is served as text/plain with a 200.

§ before the next deploy

Catch this one before Search Console does.

Search Console tells you weeks after the fact. LintPage runs 60 checks against a URL in about 30 seconds. Free, no signup.

run a full scan →