Robots.txt Checker — Parse Rules & Blocked Paths

Robots.txt is a voluntary protocol for well-behaved crawlers. It does not enforce authentication or firewall rules — anyone can request disallowed URLs directly. It guides search engine budget and reduces noisy crawling of duplicate or private UI paths.

Security through obscurity fails — never rely on Disallow alone to hide sensitive endpoints. Use authentication and network controls instead.

Each User-agent line starts a rule group. The universal * agent applies broadly; named agents like Googlebot can have overrides. Crawlers match the most specific group name they recognize.

Our parser preserves agent names and associated Allow/Disallow entries for side-by-side review.

Within a group, Allow can narrow Disallow for specific subpaths on some crawlers. Longest match wins in modern search engine implementations — verify critical paths manually in search console tools after changes.

A single Disallow: / under User-agent: * blocks the entire site for compliant bots — a frequent accidental deployment during maintenance.

Sitemap lines point crawlers to XML sitemap URLs, optionally multiple for large properties. They may appear anywhere in robots.txt, not only at the end.

We list all unique Sitemap URLs found to cross-check with your sitemap validator workflow.

Robots.txt often advertises paths admins consider sensitive — /admin, /backup, /api/internal. Attackers harvest these entries. Prefer not listing secret paths; protect them with auth regardless.

Reading robots.txt during recon is standard — treat its contents as public information.

404 on robots.txt means no file — crawlers assume full allow. 200 with empty body behaves similarly. 5xx errors may cause crawlers to pause — fix server errors promptly during launches.

We report status alongside parsed content so you distinguish missing file from empty file.

Raw field shows up to five thousand characters for diffing against version control. Very large files truncate — host extremely long rules rarely; split by subdomain instead.

Compare raw to parsed output when custom directives confuse parsers.

After generating a new file with our visual builder, verify deployment with this checker. Ensure CDN and origin both serve identical robots.txt without stale cache.

Pair with sitemap validator to confirm declared sitemap URLs resolve and validate.

Blocking CSS and JS resources harms search rendering. Wildcard typos in paths. Forgetting to remove staging Disallow after go-live. Multiple conflicting User-agent blocks duplicated across merges.

Test apex and www separately if both host robots.txt — they should redirect consistently.

We fetch once from our server location. Geo-restricted sites may block the fetch. robots.txt on non-standard ports is not supported — only default HTTPS/HTTP origin.

Parsing follows common line syntax; non-standard extensions may appear only in raw view.