Robots.txt Validator
Make sure your robots.txt isn't accidentally blocking Google from crawling your site — the #1 cause of zero organic traffic after launch.
§ what this tool checks
Rules applied to every scan.
Your robots.txt file is the first thing search engines read when they visit your site. A single syntax error or an overly broad Disallow rule can block Google from crawling your entire site — and you won't know until your traffic drops to zero. This tool validates your robots.txt for syntax errors, missing directives, and accidental blocks.
File existence and accessibility at /robots.txt
Syntax validation for User-agent and Disallow directives
Overly broad blocking rules (e.g., Disallow: /)
Sitemap reference in robots.txt
Crawl-delay directives and their impact
Conflicting or redundant rules
§ faq
Questions, answered.
What is robots.txt and why is it important?
robots.txt is a plain text file at the root of your website that tells search engine crawlers which URLs they can and cannot access. It's the first file a search engine bot reads when visiting your site. While it doesn't prevent pages from being indexed (use noindex for that), it controls crawl behavior and can accidentally block search engines from discovering your content entirely.
How do I fix a robots.txt that blocks search engines?
First, check if you have a "Disallow: /" rule that blocks all crawlers. Replace it with specific Disallow rules for only the paths you want to block (like /admin/ or /api/). Make sure your User-agent directive targets the right bots — "User-agent: *" applies to all crawlers. Always include a Sitemap directive pointing to your sitemap.xml.
Can a bad robots.txt kill my SEO?
Yes. A robots.txt with "Disallow: /" tells all search engines to stop crawling your entire site. This is the #1 cause of zero organic traffic after launch — staging environments often have this rule, and it can easily get deployed to production by accident. LintPage was built specifically to catch this kind of issue.
Should I have a robots.txt file even if I want everything indexed?
Yes. Even if you want all pages indexed, a robots.txt file should exist to reference your sitemap.xml and block paths that don't need crawling (like /api/ or /admin/). An empty or missing robots.txt is a missed opportunity to guide crawlers efficiently and can waste your site's crawl budget.
§ run the full audit
Stop guessing. Scan everything in one click.
60 automated checks across meta tags, robots.txt, Open Graph, sitemaps, headings, AI visibility, and more — free, no signup.
run a full scan →§ other tools
Check another thing.
Single-purpose inspectors for when you need to verify one thing.
01Meta Tag CheckerCheck your page title, meta description, viewport, charset, and robots tags.02AI Crawler CheckerTest if ChatGPT, Claude, and Perplexity can actually read your page — not just whether your robots.txt says they can.03Open Graph Tag PreviewCheck your Open Graph and Twitter Card tags for social media sharing.04Sitemap ValidatorValidate your sitemap.xml file format and URL count.05Heading Structure CheckerAnalyze your H1-H6 heading hierarchy for SEO best practices.06SSL Certificate CheckerCheck your SSL certificate: expiry date, issuer, and trust chain.07Redirect CheckerTrace the full redirect chain for any URL — every 301, 302, and hop.08Structured Data ValidatorValidate your JSON-LD schema markup for missing fields and errors.09Broken Link CheckerScan a page for internal links that return 404s and server errors.10Core Web Vitals CheckerCheck the lab signals behind LCP, CLS, and a fast first byte.11Security Headers CheckerScan your HTTP response headers for CSP, HSTS, and other protections.12Canonical Tag CheckerCheck your canonical tag and confirm it points where it should.