Skip to content

Cookie settings

Necessary · always on

Sign-in, security and remembering this choice. The site doesn't work without them.

Analytics

Counts visits and how pages are used, so we know what to improve. Google Analytics, Microsoft Clarity.

Marketing

Measures which ads bring visitors and lets us show ads to people who visited. Meta, Google Ads.

Robots.txt tester: is this URL blocked?

Check whether Googlebot, Bingbot or any crawler may fetch a URL, and see the exact robots.txt rule that decides it.

FreeNo account neededRFC 9309 matching

How it decides

Group
Most specific user agent
Rule
Longest match wins
Tie
Allow wins
Wildcards
* any · $ end of URL
Fetch
First 500 KiB, like Google

We fetch /robots.txt from the URL's host and read the first 500 KiB, as Google does.

01

Allowed or blocked, for the crawler you pick.

02

The exact rule that decides it, highlighted in the file.

03

Which user-agent group applied, and why.

04

Every Sitemap: line the file declares.

01How it works.

  1. 01

    Fetch or paste robots.txt

    Enter a URL and we fetch robots.txt from its host (the first 500 KiB, as Google does). Or paste a draft file to test it before it goes live; that runs only in your browser.

  2. 02

    Pick the crawler's group

    The crawler follows the most specific user-agent group that matches it (Googlebot-Image falls back to Googlebot), else the * group. Groups for the same crawler are merged.

  3. 03

    Longest rule wins

    Of the Allow and Disallow rules that match the path, the longest one decides; on a tie, Allow wins. * matches anything and $ marks the end of the URL.

02Questions.

It uses the most specific rule: the one with the longest path that matches the URL. If an Allow and a Disallow rule are equally long, Allow wins. The order of the lines in the file doesn't matter.

More free tools.

All tools

Connect Search Console to see this for every page, daily.

Daily Search Console data for positions, clicks and CTR on every page, with alerts. Free plan, no card.