Robots.txt tester: is this URL blocked?
Check whether Googlebot, Bingbot or any crawler may fetch a URL, and see the exact robots.txt rule that decides it.
FreeNo account neededRFC 9309 matching
How it decides
- Group
- Most specific user agent
- Rule
- Longest match wins
- Tie
- Allow wins
- Wildcards
- * any · $ end of URL
- Fetch
- First 500 KiB, like Google
Allowed or blocked, for the crawler you pick.
The exact rule that decides it, highlighted in the file.
Which user-agent group applied, and why.
Every Sitemap: line the file declares.
01How it works.
- 01
Fetch or paste robots.txt
Enter a URL and we fetch robots.txt from its host (the first 500 KiB, as Google does). Or paste a draft file to test it before it goes live; that runs only in your browser.
- 02
Pick the crawler's group
The crawler follows the most specific user-agent group that matches it (Googlebot-Image falls back to Googlebot), else the * group. Groups for the same crawler are merged.
- 03
Longest rule wins
Of the Allow and Disallow rules that match the path, the longest one decides; on a tie, Allow wins. * matches anything and $ marks the end of the URL.
02Questions.
It uses the most specific rule: the one with the longest path that matches the URL. If an Allow and a Disallow rule are equally long, Allow wins. The order of the lines in the file doesn't matter.
No. Disallow stops crawling, not indexing: a blocked URL can still be indexed from links, without its content. To keep a page out of search results, let it be crawled and use a noindex meta tag or header instead.
A 4xx answer (such as 404) means there are no restrictions: crawlers may fetch everything. A 5xx answer or an unreachable server makes Google treat the whole site as disallowed until robots.txt works again. The tester shows which case applies.
Googlebot covers Google Search. Test Bingbot for Bing, or type any product token (for example GPTBot or Googlebot-Image) as a custom user agent. Only the token matters, not the full user-agent string.
It follows RFC 9309, the robots.txt standard Google co-authored, plus Google's documented behaviour (500 KiB limit, user-agent fallback, common misspellings). We fetch robots.txt as SerplyzeBot; a site that serves crawlers a different file can answer differently.
More free tools.
All tools- Reads public sitemaps
Website page counter
List every URL a site's XML sitemaps contain, with counts per sitemap and per folder, and download them as CSV.
Open the tool - Runs in your browser
Search Console regex tester
Check a Custom (regex) filter against sample queries or URLs with RE2, the engine Search Console uses. 15 presets included.
Open the tool - File stays in your browser
CTR curve and visibility score calculator
Build your own CTR-by-position curve from a Search Console export, find queries below it and get a visibility score.
Open the tool - Runs in your browser
SERP snippet preview
Measure a title and meta description in pixels, preview them at desktop and mobile widths and see where they get cut.
Open the tool - Fetches up to 20 alternates
Hreflang checker
Check a page's hreflang set: every alternate is fetched and checked for return links, x-default, ISO codes and canonicals.
Open the tool - Follows up to 10 hops
Redirect checker
Trace a URL's redirects hop by hop with status codes and timings, and spot chains, loops, http/https mixes and meta refreshes.
Open the tool
Connect Search Console to see this for every page, daily.
Daily Search Console data for positions, clicks and CTR on every page, with alerts. Free plan, no card.