Robots.txt Tester
SEO Tools
Test which URLs your robots.txt actually allows or blocks. Paste your rules and a list of paths; pick a user-agent; get per-path verdicts with the matching rule highlighted.
Runs entirely in your browserAbout Robots.txt Tester
Reading a robots.txt by eye and predicting which URLs it'll block is harder than it sounds. The longest-match-wins rule, wildcard semantics, and per-user-agent group precedence all combine in ways that make manual verification error-prone. A 'short test' against one URL works; a comprehensive check across a directory takes a real tool.
This tester uses the same precedence logic Google's official robots tester uses: select the most specific User-agent group, find the matching rule with the longest path pattern, and tie-break in favor of Allow over Disallow. Paste your robots.txt, paste a list of URL paths (one per line), and pick the user-agent string you want to evaluate. The tool returns a verdict for each path with the matching rule and a one-line explanation. Use it to catch regressions before you deploy a new robots.txt and to debug 'why isn't Google crawling this page' issues. All evaluation is local to your browser.
How to use
- 1
Paste your robots.txt
Drop the full contents of your robots.txt into the left textarea. The parser handles comments, multiple user-agent groups, Allow/Disallow, Crawl-delay, and Sitemap lines.
- 2
List the paths you want to test
One path per line in the middle textarea. Include the leading slash. You can paste a real list from your sitemap or just type representative samples.
- 3
Pick the user-agent
Defaults to `Googlebot`. Change to `Bingbot`, `DuckDuckBot`, `*`, or any custom user-agent string you need to test against. The most specific matching group wins.
- 4
Read the verdicts
Each path shows Allowed or Disallowed, the rule that matched (if any), and a one-line reason. Use this to verify before deploying a robots.txt change.
Examples
Sample rules, sample paths
Input
User-agent: *
Disallow: /admin/
Allow: /api/public/
Disallow: /api/Output
/admin/users → Disallowed (matched Disallow: /admin/)
/api/v1/users → Disallowed (matched Disallow: /api/)
/api/public/health → Allowed (matched Allow: /api/public/ — longer wins)Frequently asked questions
What's the longest-match-wins rule?+
When multiple rules could match a path, the rule with the longest path pattern wins. `Allow: /api/public/` beats `Disallow: /api/` for the URL `/api/public/health` because the Allow pattern is more specific (longer).
How are wildcards handled?+
`*` matches any character sequence. `$` matches end-of-URL. So `Disallow: /*.pdf$` blocks all PDFs. `Disallow: /draft/*` blocks anything under `/draft/`.
What does 'no matching group' mean?+
Your robots.txt has groups for specific user-agents but no `User-agent: *` fallback, and the user-agent you tested doesn't match any group. Default behavior in that case is Allowed.
Does the order of rules in a group matter?+
No. Google's robots evaluator uses longest-match-wins regardless of rule order in the file. Putting Allow before Disallow (or vice versa) doesn't change which rule wins.
Why is a path I expected to be blocked showing as Allowed?+
Most common cause: a more specific Allow rule beats your Disallow. Second most common: the User-agent group you're testing doesn't match — try changing the user-agent or adding a `*` group to your robots.txt. Use the 'Matched rule' column to see what won.
Does this match Google's official tester exactly?+
It implements the same documented precedence rules. There may be edge cases (e.g., percent-encoded paths) where Google's behavior differs from our parser. For mission-critical changes, verify with Google's official robots tester in Search Console.
Related tools
Meta Tag Generator
Build a complete set of HTML meta tags — basic SEO, Open Graph, Twitter Cards, viewport — with character-count warnings.
SERP Snippet Preview
Preview how your title, URL, and meta description render in Google search results — desktop and mobile.
hreflang Generator
Build the hreflang link tag set for multilingual or multi-regional sites with BCP 47 validation.
Readability Score
Score your content with six readability formulas — Flesch, Gunning Fog, SMOG, Coleman-Liau, ARI.
Keyword Density Analyzer
Analyze 1-, 2-, and 3-word phrase frequency and density in your content with stopword filtering.
Robots.txt Generator
Build a valid robots.txt visually with user-agent groups, allow/disallow rules, and sitemap URLs.