darinor
Tools

AI Crawler Policy Checker

Enter a domain to see how its robots.txt treats AI training, search and user-triggered crawlers, and whether its llms.txt is well formed. In fetch mode the domain you enter is sent to this site's server, which fetches those two files; nothing is stored. Paste mode stays entirely in your browser.

A full URL works too; only the hostname is used. Only /robots.txt and /llms.txt are requested.

Fetch mode sends the domain to this site's server, which requests the two files. It is not stored or logged beyond the request.

About this tool

AI Crawler Policy Checker

Enter a domain and the tool requests two files from it, /robots.txt and /llms.txt, and nothing else. The domain you enter is sent to this site's server, which fetches those two files on your behalf; nothing is stored or logged beyond the request itself. If you would rather not send anything, switch to paste mode: you paste the file contents and the analysis runs entirely in your browser with no network request.

The robots.txt analysis follows RFC 9309 in plain terms. A crawler looks for the group that names it; if there is none, it falls back to the wildcard (*) group; if there is neither, everything is allowed. A missing robots.txt (a 404) means no restrictions, while a server error means crawlers must assume the whole site is off limits. The longest matching rule wins, and Allow beats Disallow when two rules match equally. Each bot gets a verdict (allowed, blocked, partly blocked, or no rule) and the source of that verdict.

Bots are grouped by purpose. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content that may be used to train models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so AI search products can cite them. User-triggered fetchers such as ChatGPT-User and Claude-User retrieve a page when a person asks an assistant to read it. Blocking one purpose does not block the others, and the tool's summary line says which are blocked and which are allowed.

The limits are real. The tool reports what your files say, not whether any crawler obeys them: robots.txt is advisory and enforces nothing. The bot list is reviewed periodically and vendors rename and add crawlers, so check each vendor's documentation. llms.txt is a proposal rather than a standard, so a missing file is informational, not an error.

When to use it

  • Block model training but stay in AI search

    Paste your robots.txt and confirm that GPTBot, ClaudeBot and CCBot are blocked while OAI-SearchBot, Claude-SearchBot and PerplexityBot are still allowed.

  • Audit a competitor's or client's published policy

    Enter their domain to see which AI crawlers they allow, and whether their llms.txt exists and is well formed.

  • Catch a robots.txt mistake before it ships

    Paste a draft to find soft-404 HTML served as robots.txt, invalid lines, misspelled bot names and rules that block more than intended.

Questions

Does blocking GPTBot remove me from ChatGPT search?

Not by itself. OpenAI documents separate crawlers for separate purposes: GPTBot relates to training, OAI-SearchBot relates to ChatGPT search results, and ChatGPT-User fetches pages when a user asks. Blocking training and blocking search are separate controls, so check each token and read the vendor's documentation, because vendors change how these crawlers work.

Is my domain stored?

No. In fetch mode the domain is sent to this site's server, which requests the two files and returns them to you; it is not stored or logged beyond the request. In paste mode nothing is sent anywhere. See the privacy page for details.

Why does the tool use Cloudflare Turnstile?

Fetch mode makes this site's server request files from other websites, so a Turnstile check keeps automated abuse out. Paste mode does not use it.

Does robots.txt actually stop AI crawlers?

Only the ones that choose to honor it. robots.txt is a voluntary convention from RFC 9309. Reputable vendors document that their crawlers obey it, but it is not access control; to enforce a policy you need server-side or network controls.

What happens if my robots.txt returns an error?

A 404 or similar means crawlers assume no restrictions. A 5xx error or a site that is unreachable is treated by crawlers as a complete disallow under RFC 9309. A page that returns HTML with a 200 status (a soft 404) contains no valid rules, so it counts as no restrictions.

Do I need an llms.txt file?

No. llms.txt is a community proposal for giving language models a curated summary of a site, and no crawler is required to read it. The tool checks its structure if you have one and treats a missing file as informational.