Robots.txt Checker

Server assisted New Website & HTTP Diagnostics

Reads the robots.txt at the root of a site through the Mutqan server, or a file you paste, and lists its groups, rules, Sitemap lines and syntax problems. Test any list of paths for a crawler such as Googlebot: matching follows RFC 9309 (longest match wins, Allow wins a tie, * and $ wildcards) and each result names the line that decided it.

Privacy: The Mutqan server sends the small, fixed test requests described on this page to the host you name, and nothing is stored. Use it only against systems you are authorised to test.

What is robots.txt?

robots.txt is a plain-text file at the root of a host that tells crawlers which paths they may fetch, as groups of Allow and Disallow rules addressed to User-agent tokens. RFC 9309 standardised it in 2022: the most specific matching rule wins, a file that answers 4xx means no restrictions, and a 5xx means nothing may be crawled. It controls crawling, not indexing, and obeying it is voluntary.

Security note: robots.txt is public: a Disallow line that names a private admin path advertises it to anyone who reads the file. Protect private areas with authentication, not with robots.txt.

How to use Robots.txt Checker

  1. Enter any URL of the site (the checker reads /robots.txt from its root), or paste the file into the second field to check it without fetching.
  2. Type the crawler to test — a token such as Googlebot or Bingbot, or * for crawlers without their own group.
  3. List the paths or URLs to test, one per line, and press Check robots.txt.
  4. Read the Path test table: Allowed or Disallowed, the deciding rule and its line. Fix what the Errors and Warnings lists point to, then check again.

Robots.txt Checker features

  • Fetches /robots.txt from the root of any public site through the Mutqan server, or parses a file you paste without sending it anywhere
  • Tests a list of paths or full URLs for one crawler token (Googlebot, Bingbot, *) and names the rule and line that decided each result
  • RFC 9309 matching: the crawler's own group replaces the "*" group, the longest path wins, Allow wins a tie, * and $ wildcards, percent-encoding normalised
  • Applies the status-code rules crawlers follow: a 4xx means no restrictions and a 5xx makes everything disallowed; Googlebot treats 429 the same way, so the checker does too when the crawler you test is a Google one
  • Lists every User-agent group with its Allow and Disallow counts and Crawl-delay, and every Sitemap line
  • Reports syntax errors and suspicious lines with line numbers: rules before any User-agent, relative Sitemap URLs, misspelt directives, paths without a leading slash
  • Shows the fetch itself: final URL after redirects, HTTP status, Content-Type, size, resolved address and time

Inputs and outputs

Inputs

  • Site URL — any address on the site; the checker requests /robots.txt at the root of that scheme, host and port.
  • Or a pasted robots.txt — parsed in your browser with nothing fetched; the URL, if also given, only identifies the host for the path test.
  • User agent to test — a crawler product token (Googlebot, Bingbot, *) or a full User-Agent string, from which the token is read.
  • Paths or URLs to test — one per line, up to 200; a query string counts, a #fragment is ignored.

Outputs

  • Summary — the HTTP status of the fetch and how crawlers treat it, the number of groups, rules and Sitemap lines, the group used for your crawler, and the error and warning counts.
  • Path test — Allowed or Disallowed for each path, with the deciding rule and its line number.
  • Groups — every User-agent group with its Allow and Disallow counts, Crawl-delay and first line.
  • Sitemaps, Errors and Warnings — the declared sitemap URLs and the problems found, each with its line number.
  • Fetch details — requested and final URL, status, Content-Type, bytes downloaded, resolved address and timing.

Robots.txt Checker example

A Googlebot group that replaces the "*" rules

Input:

Pasted file: User-agent: * / Disallow: /cart/ / Disallow: /search / Allow: /search/help / Crawl-delay: 5 — User-agent: Googlebot / Disallow: /tmp/ / Disallow: /*.pdf$ — Sitemap: https://www.example.com/sitemap.xml
User agent: Googlebot · Paths: /cart/checkout, /search?q=shoes, /docs/guide.pdf, /tmp/report

Output:

Groups 2 · Rules 5 · Sitemaps 1 · Group used googlebot (line 8) · Errors 0 · Warnings 1
/cart/checkout → Allowed (no matching rule: the * group does not apply to Googlebot)
/search?q=shoes → Allowed (no matching rule)
/docs/guide.pdf → Disallowed — Disallow: /*.pdf$ (line 10)
/tmp/report → Disallowed — Disallow: /tmp/ (line 9)
Warning: Line 6: Crawl-delay is not part of RFC 9309 — Googlebot ignores it; Bing honours it.

How to read the Robots.txt Checker results

Allowed and Disallowed describe crawling only. Disallowed means a compliant crawler will not request the URL; it does not hide the URL, protect it or remove it from search results.

"No matching rule" means the crawler's group has no rule for that path, so it is allowed. "Group used: —" means the file has neither a group naming the crawler nor a "*" group, so nothing applies to it.

A 4xx answer for robots.txt (usually 404) makes every path allowed; a 5xx or a timeout makes every path disallowed until the file can be read, and Googlebot treats 429 the same way as a 5xx. A site that should be crawled must answer 200 or 404 for /robots.txt, never an error.

Errors are lines crawlers ignore (no colon, a rule before any User-agent line, a relative Sitemap URL). Warnings are lines crawlers read, but probably not as the author meant (a path without a leading "/", an empty Disallow, a misspelt directive, Crawl-delay).

When to use Robots.txt Checker

  • Before a release, confirm that a new Disallow blocks only the admin or staging paths it was written for and not the product pages next to them.
  • Find out why Search Console reports "Blocked by robots.txt": test the URL as Googlebot and read the deciding line.
  • After moving a site to a new host or to HTTPS, check that the new origin serves its own robots.txt and that its Sitemap lines point to the new addresses.
  • Review a file that gives specific crawlers their own groups and check which group each token falls into.

Troubleshooting

  • Many "not a field: value line" errors and a text/html Content-Type: the server answers /robots.txt with an HTML page, often an error page or an app shell sent with status 200. Serve a plain-text file at that path.
  • HTTP 401 or 403 for robots.txt: crawlers treat it as "no restrictions", which is rarely what the owner wants. The file must be publicly readable.
  • A path you expect to be disallowed shows Allowed: check letter case (paths are case-sensitive), a missing leading "/", and whether the crawler has its own group, which replaces every "*" rule.
  • A refusal such as "resolves to an address the Mutqan server may not connect to" or "Port … is not allowed": the host is private, reserved or on a port outside 80, 443, 8080, 8443 and the 3000–3999, 5000–5999 and 8000–8999 ranges. Paste the file instead.
  • The fetch ends on another host after redirects: crawlers apply the file they end on to the original host. Serve robots.txt directly on each host when the rules should differ.

Frequently asked questions about Robots.txt Checker

Why is a path allowed although a Disallow line seems to match it?

RFC 9309 applies the most specific rule: the one with the longest matching path, and Allow wins when an Allow and a Disallow are equally long, so "Allow: /search/help" beats "Disallow: /search" for /search/help. Rules also belong to groups: when the file has a group for Googlebot, Googlebot ignores the "*" group entirely. The Path test table names the line that decided each result.

What do crawlers do when robots.txt returns 404 or 500?

A 4xx answer (404, 403, 401) means the file is unavailable and crawlers may fetch every path (RFC 9309 §2.3.1.3). A 5xx answer or no answer means it is unreachable and crawlers must assume every path is disallowed (§2.3.1.4). RFC 9309 counts 429 (Too Many Requests) among the 4xx answers, but Googlebot treats it the same way as a 5xx, so the checker applies the 5xx rule to 429 when the crawler you test is a Google one and the RFC rule otherwise.

Does Disallow keep a page out of Google?

No. robots.txt controls crawling, not indexing: a disallowed URL can still be listed from links that point to it, just without its content. To keep a page out of results, let it be crawled and serve a noindex directive (meta robots or X-Robots-Tag) — a crawler that may not fetch the page never sees the noindex.

Which file does the checker fetch when I enter a page URL?

Always /robots.txt at the root of the same scheme, host and port: https://shop.example.com/a/b reads https://shop.example.com/robots.txt. Each origin has its own file, so www.example.com and example.com, or http and https, can answer differently.

Are Crawl-delay and other non-standard lines checked?

Crawl-delay is parsed and shown per group, with a warning that it is not part of RFC 9309 (Googlebot ignores it; Bing honours it; Yandex uses the crawl-rate setting in Yandex Webmaster instead). Host, Clean-param and similar lines are reported as extensions most crawlers ignore, and misspellings that Google tolerates, such as "Dissallow", are read with a warning.

Technical notes

Parsing and matching follow RFC 9309, the Robots Exclusion Protocol (2022). Consecutive User-agent lines open a group; a crawler uses the groups whose product token matches its own, case-insensitively — the longest matching token wins — and falls back to the "*" groups only when none matches. Rules are compared on percent-normalised paths, * matches any sequence of characters, a trailing $ anchors the end, and /robots.txt itself is always allowed.

Fetch outcomes follow RFC 9309 §2.3.1: a 2xx file is parsed, up to five redirects are followed (the Mutqan server follows at most five), a 4xx means the file is unavailable and nothing is restricted, and a 5xx means it is unreachable and everything is disallowed. RFC 9309 counts 429 among the 4xx answers; Google's crawlers treat it as a server error instead, so the checker applies the 5xx rule to 429 only when the crawler token is a Google one. Only the first 500 KiB are parsed, the minimum the RFC asks crawlers to read.

Sitemap lines are global — they belong to no group — and must be absolute URLs, as the sitemaps.org protocol requires.

Limitations and methodology

The file is fetched once, from the Mutqan server on Cloudflare's network, with the fixed Mutqan User-Agent. A site that serves a different robots.txt by User-Agent, country or IP range may show real crawlers another file.

At most 512 KiB is downloaded and the first 500 KiB parsed. Rules past that point are ignored by crawlers too, so they are not tested.

Results show how a crawler that implements RFC 9309 reads the file. Individual crawlers differ in details — some ignore * and $, Googlebot ignores Crawl-delay — and the checker cannot tell whether a crawler obeys the file at all.

Security and privacy

The paths you test never leave your browser: matching runs there, on a pasted file or on the copy the Mutqan server fetched. For a fetch, the server receives only the address of /robots.txt on that host, requests it once and passes the answer back without keeping it.

The request to the site carries a fixed set of headers and none of your cookies or credentials. Check only sites you are allowed to test.

References

  • RFC 9309: Robots Exclusion ProtocolIETF · 2022The robots.txt standard: groups of User-agent, Allow and Disallow lines, longest-match precedence, the * and $ special characters, and what crawlers do when the file answers 4xx or 5xx.
  • How Google Interprets the robots.txt SpecificationGoogle for DevelopersHow Google's crawlers read robots.txt: the 500 KiB limit, redirects, status codes (429 treated as a server error), group selection and the rules Google tolerates or ignores, such as Crawl-delay.