Robots.txt is a single-line-of-text file most sites edit twice a year and then forget. One stray Disallow: / in staging that ships to production silently removes your entire site from Google.
This tool fetches the live robots.txt at the origin, parses it using the same longest-match rule Google uses, then tells you exactly which line allows or blocks the URL you tested for the crawler you picked.
Where this tool actually helps
Confirm a launch didn't ship a staging robots.txt
Paste your homepage URL. If Googlebot is blocked on /, you found the bug in 10 seconds.
Decide whether to allow GPTBot or PerplexityBot
Test the same URL as GPTBot vs Googlebot to see if your AI crawler policy actually matches your intent.
Debug a missing sitemap in GSC
The sitemap discovery pane lists every Sitemap: line the file declares. If yours is missing, Google won't find it.
How to read the result
Green means allowed, red means blocked. The matched rule tells you which line is in charge. When no rule matches, the URL is allowed by default.
User-agent matching is substring-based, exactly as Google specifies. "Googlebot-Image" matches a "Googlebot" block.
Common mistakes we see
- Using robots.txt to hide sensitive URLs. It doesn't. Use auth or noindex — robots.txt is a public file.
- Blocking CSS or JS. Google needs both to render your page correctly.
- Copying an example robots.txt without changing the sitemap line to your own domain.
FAQs
- Does this crawl the URL?
- No. It only fetches /robots.txt at the origin and matches your test path against the parsed rules.
- Which matching rule does it use?
- Google's rule: the longest matching Allow or Disallow path wins. Wildcards * and end-anchor $ are supported.
- Why is my URL allowed even though robots.txt has Disallow?
- A more specific Allow beats a broader Disallow. The tool shows you the exact matched rule.
A note from Nimitt
A tool gives you a number. A person tells you what to do with it. If you want a straight answer on your site, send it over. I read every one myself.