Real precedence
Longest matching user agent wins and * is a fallback, not a competitor. Within a group the longest path pattern wins, and Allow beats Disallow on a tie.
Free tool · No signup
Test a URL against robots.txt
This robots.txt tester applies the rules the way a crawler does and names the line responsible, replacing the tester Google retired from Search Console.
robots.txt tester Free
- Any user agent
- Shows the deciding rule
- No signup
TEST RESULT
Result
WHY
GROUPS
Rules in this file
| User agents | Rules | Crawl-delay |
|---|
ROBOTS.TXT
CHECKING MORE THAN ONE URL?
Test a thousand paths in a loop.
You just checked one URL against one crawler. The Web Scraping API returns the same robots.txt and page data as JSON, so you can run the check across every URL in your sitemap on a schedule.
WHAT THIS TESTER DOES
Names the rule
Every verdict cites the directive and line number that produced it, so you can go and change the right line.
Any user agent
Test Googlebot, Bingbot, GPTBot, or a name you type yourself, because groups apply per crawler.
ROBOTS.TXT TESTER GUIDE
Since Search Console dropped its tester.
Google removed the robots.txt Tester from Search Console, leaving people without an easy way to check a specific URL. This guide covers how the rules actually resolve, what robots.txt does and does not do, and the mistakes that survive a casual read.
How robots.txt rules actually resolve
Most confusion about robots.txt comes from assuming the rules are read top to bottom like a script. They are not. RFC 9309 defines a specific order.
The crawler picks one group
A crawler finds the group whose user-agent value is the longest match for its own name, and uses only that group. User-agent: * is a fallback used when nothing else matches, not a rule that stacks on top of a named group. This is why adding User-agent: GPTBot with a single line silently discards every rule you wrote under * for that crawler.
The longest path pattern wins
Within the chosen group, the most specific matching pattern decides, not the first or last one written. Disallow: /admin/ and Allow: /admin/public/ together permit /admin/public/page, because the allow pattern is longer.
Allow wins ties
When an allow and a disallow match at exactly the same length, the allow wins. And Disallow: with nothing after it matches no path at all, which is the conventional way to say "everything is permitted".
Patterns support * for any sequence and $ to anchor the end, so Disallow: /*.pdf$ blocks PDFs but not /file.pdf.html.
Blocking a URL is not removing it
This is the misunderstanding that causes real damage. robots.txt asks a crawler not to fetch a URL. It does not ask a search engine to forget the URL exists.
A blocked page can still appear in results, usually with no description, because other pages link to it and the engine knows the address without having read the content. Worse, blocking a page makes its noindex directive unreadable, so a URL you block to hide can become harder to remove than one you leave open.
To keep a page out of an index, allow crawling and serve noindex. To keep a crawler off your server, block it. They are different jobs.
Mistakes this robots.txt tester catches
A named group that discards your wildcard rules. The most common surprise, and the tester shows which group applied.
Blocking assets. Disallowing /assets/ or /static/ stops a renderer fetching the CSS and JavaScript a page needs, so it is judged on a broken version of itself.
Assuming subdomains inherit. Every host serves its own robots.txt. Rules on the apex do nothing for docs. or app..
A trailing slash that changes the meaning. Disallow: /admin blocks /administrator too; Disallow: /admin/ does not.
Treating it as security. robots.txt is a public file listing the paths you would rather nobody visited. It is a request to well-behaved crawlers, not access control.
robots.txt questions
Where is the robots.txt tester in Google Search Console?
Google retired it. Search Console still shows the robots.txt files it fetched under Settings, but no longer lets you test a URL against them, which is what this tool does.
How do I test if a URL is blocked?
Enter the site and the path. The tester applies the rules as a crawler would and reports the verdict with the rule and line responsible.
Which rule wins when Allow and Disallow both match?
The longest matching pattern. At equal length, Allow wins. An empty Disallow: matches nothing.
Does robots.txt stop a page being indexed?
No. It asks crawlers not to fetch it. A blocked page can still be listed if others link to it. Use noindex to keep a page out of an index.
Does robots.txt cover subdomains?
No. Each host has its own file, so test each separately.
Know which line is responsible
A verdict is only useful if you can act on it. This robots.txt tester names the directive and line behind every result.
Crawl permission is one signal among several that decide whether a page is fetched and understood.
Check AI crawlers too
See every AI bot's verdict at once with the AI Crawler Access Checker.
Check your sitemap
robots.txt declares where it lives. Validate it with the XML Sitemap Checker.
Check the page itself
Crawlable is not the same as sound. Run the SEO Page Inspector.