Robots.txt tester: is this URL blocked for Googlebot?

Enter a full page URL, pick a crawler and see whether robots.txt allows it, which rule decides it, and how other crawlers are treated.

How robots.txt matching works

The crawler looks for the group that names it most specifically, like Googlebot. If none matches, it uses the * group.
Inside that group, the longest matching path rule wins.
If an Allow and a Disallow rule are the same length, Allow wins.
* matches any characters and $ marks the end of the URL.

The tester applies the same logic, so you can see exactly why a URL is allowed or blocked.

Disallow: / left over from a staging site, blocking the whole site.
Blocking CSS or JavaScript folders, so Google can't render pages properly.
Blocking parameters too broadly, hiding real product or filter pages.
Using robots.txt to remove pages from Google. It stops crawling, not indexing. Use noindex instead.

To check AI crawlers specifically, use the AI crawler checker.

Does blocking a URL in robots.txt remove it from Google?

No. Google can still index a blocked URL if other pages link to it. Use a noindex tag and let Google crawl the page to remove it.

Is robots.txt case sensitive?

Paths are case sensitive. User-agent names are not.

Where must robots.txt live?

At the root of each host, like example.com/robots.txt. A subdomain needs its own file.

Free tools find the problems. These services fix them.

Send me your site. You'll get a clear plan within a day.

Start a project WhatsApp me