About the Robots & Crawler Check
This tool shows how robots see a page. It reads the robots.txt of the site, works out which group of rules applies to each individual crawler, and tells you whether that crawler may fetch the URL you entered. On top of that it reads the directives that live on the page itself, the X-Robots-Tag response header and the meta robots tag, which decide whether the page may be indexed once it has been fetched.
Those two things get confused constantly, and the difference matters. Robots.txt controls crawling, not indexing. A URL that is disallowed in robots.txt can still turn up in search results as a bare link, because a search engine is allowed to know a URL exists without ever fetching it. Conversely a noindex tag only works if the crawler is allowed to fetch the page and read it. Block the page in robots.txt and add a noindex, and you get the worst of both: the crawler never sees the noindex, so it never learns that you wanted the page out.
There is no score and no pass or fail here, because there is no universally correct answer. A staging site should be closed. A marketing site should be open. Plenty of people want to be in Google but not in a training set, and that is a perfectly coherent position that needs two different rules. The tool reports which rules apply to whom so you can check the outcome against what you actually intended.
Crawlers are listed individually rather than lumped together, because vendors run several under different names. OpenAI uses GPTBot for training, OAI-SearchBot for its search index and ChatGPT-User for pages a person asks ChatGPT to open. Google-Extended controls Gemini training and has no effect on Google Search. Applebot-Extended is Apple Intelligence training while plain Applebot still crawls for Siri and Spotlight. Blocking one of a vendor's tokens does not block the others.
How to use it
- Enter any URL. Enter a specific page, not just the domain, if you want to know about that page.
- Read the three verdicts at the top for search engines, AI search and AI training.
- Check the page panel for X-Robots-Tag and meta robots, which are what control indexing.
- Scan the per-crawler table to see which robots.txt rule decided each bot's outcome.
- Look at the decided-by column: it names the exact rule and the user-agent group it came from.
- Open the raw robots.txt if a result surprises you, and compare it with the parsed groups.
- Read the notes at the bottom for interactions that are easy to miss, like a noindex on a disallowed URL.
Examples
- https://example.com/
- example.com/admin/settings
- https://example.com/search?q=test
- https://staging.example.com/
Crawling and indexing are different controls
Almost every robots problem comes down to using the wrong one of these two. They live in different places and they answer different questions.
- robots.txt: may this crawler request this URL at all. It is fetched once per site and cached, and it never mentions indexing.
- X-Robots-Tag: an HTTP response header that controls indexing. It works for any file type, which makes it the only option for PDFs, images and other non-HTML responses.
- meta robots: the same directives as X-Robots-Tag but inside the HTML head, so it only works for HTML pages.
- canonical: not an access control at all. It tells a search engine which URL to prefer when several show the same content.
- The trap: a noindex is only obeyed if the crawler is allowed to fetch the page and read it. Blocking the URL in robots.txt hides the noindex from the crawler.
How the robots.txt rules are interpreted
Robots.txt matching has rules that surprise people, and this tool applies them the way the standard and the major crawlers do.
- The most specific rule wins, meaning the longest matching path pattern, not the first or the last line in the file.
- When an Allow and a Disallow match with the same length, Allow wins.
- An asterisk matches any sequence of characters, and a dollar sign anchors the match to the end of the URL.
- A crawler uses the group that names it, and only falls back to the wildcard group if no group names it. Rules in the wildcard group do not add to a named group, they are replaced by it.
- An empty Disallow imposes no restriction. Disallow with a single slash blocks the whole site.
- Paths are case sensitive, but user-agent names are not.
What the status code of robots.txt itself does
The response code of the robots.txt file changes the outcome for the whole site, and the behaviour is not what most people assume.
- 2xx: the rules in the file apply as written.
- 404 or another 4xx: crawlers read this as no restrictions at all, so the entire site is open.
- 429 or any 5xx: Google treats this as a temporary instruction to stay away from the whole site. A broken server can therefore quietly stop crawling.
- Served as HTML instead of plain text: usually a catch-all route answering rather than a real file. Crawlers still try to parse it, with unpredictable results.
- Over 500 KiB: Google stops reading at that point, so any rules after it are ignored.
About llms.txt
llms.txt is a proposed convention: a markdown file at the root of a site that points language models at the content you consider important, in a clean form. It is not part of any standard, no crawler is obliged to read it, and it grants no permission and blocks nothing. The tool reports whether the file is there and how large it is, so you know where you stand. If you want to actually control AI crawlers, robots.txt is where that happens.
Frequently asked questions
Why does a page I blocked in robots.txt still show up in Google?
Because robots.txt blocks crawling, not indexing. Google can learn a URL exists from links elsewhere and list it without ever fetching it, usually with no description. To keep a page out of the index you have to let it be crawled and serve a noindex, or put it behind authentication.
Should I block AI crawlers?
That is a business decision, not a technical one, which is why this tool does not grade it. Blocking training crawlers protects your content from being used for models. Allowing answer engines like ChatGPT Search and Perplexity keeps you visible where a growing share of people now ask questions. The two are separate tokens, so you can block one and allow the other.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended only governs whether your content is used to train and ground Gemini. Google Search indexing is controlled by Googlebot. They are separate tokens and blocking one has no effect on the other. Applebot and Applebot-Extended split the same way.
Why does the tool check the URL I gave rather than just the domain?
Because robots.txt rules are per path. The homepage can be wide open while a search or admin path is blocked, and the reverse happens too. Robots.txt and llms.txt are always read from the root of the site, but the allow and disallow verdicts are calculated for the exact path you entered.
The verdict says mixed. What does that mean?
It means crawlers in that category do not all get the same answer, usually because the site has a rule group naming some bots specifically while the rest fall back to the wildcard group. The per-crawler table shows exactly who gets what and which rule decided it.
Do all crawlers actually obey these rules?
Robots.txt is voluntary. The major search engines and the well-known AI crawlers honour it, and the SEO tools generally do too. Scrapers that want your content regardless will ignore it entirely. If a page genuinely must not be read, robots.txt is the wrong tool and authentication or an IP block is the right one.