Free tool
AI crawler check: which AI crawlers your robots.txt lets in.
Enter your domain. You see, for 13 AI crawlers, whether your robots.txt lets each one read your home page, what each is for according to the company that runs it, and which ones AI answers depend on.
Check a site
Sample: taskmoor.example lets 7 of 7 answer crawlers read its home page, and blocks 3 of 6 training, data and other crawlers.
| Crawler | Kind | Home page | Why |
|---|---|---|---|
| OAI-SearchBotOpenAI | Answers | Allowed | Allow: / (User-agent: *) |
| ChatGPT-UserOpenAI | Answers | Allowed | Allow: / (User-agent: *) |
| GPTBotOpenAI | Training or data | Blocked | Disallow: / (User-agent: GPTBot) |
| Claude-SearchBotAnthropic | Answers | Allowed | Allow: / (User-agent: *) |
| Claude-UserAnthropic | Answers | Allowed | Allow: / (User-agent: *) |
| ClaudeBotAnthropic | Training or data | Allowed | Allow: / (User-agent: *) |
| PerplexityBotPerplexity | Answers | Allowed | Allow: / (User-agent: *) |
| Perplexity-UserPerplexity | Answers | Allowed | Allow: / (User-agent: *) |
| GooglebotGoogle | Answers | Allowed | Allow: / (User-agent: *) |
| Google-ExtendedGoogle | Training and Gemini answers | Blocked | Disallow: / (User-agent: Google-Extended) |
| Applebot-ExtendedApple | Training or data | Allowed | Allow: / (User-agent: *) |
| CCBotCommon Crawl | Training or data | Blocked | Disallow: / (User-agent: CCBot) |
| BytespiderByteDance | Undocumented | Allowed | Allow: / (User-agent: *) |
No sign-up. Up to 5 checks an hour from one connection.
Reading it
How to read your result
Each row says whether one crawler may fetch your home page under your robots.txt, and the rule that decided it. The sample above is a fictional site, taskmoor.example, with this robots.txt:
User-agent: * Allow: / Disallow: /app/ User-agent: GPTBot User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: /
It lets 7 of 7 answer crawlers in, and blocks 3 of 6 others: GPTBot and CCBot by name, and Google-Extended, which also keeps its pages out of grounding in Gemini Apps. Everything else falls back to the User-agent: * group, which allows the home page.
- Allowed means robots.txt doesn't stop that crawler at your home page. Deeper pages can still be blocked by a longer rule.
- Blocked means a rule stops it. The Why column shows the rule and the group it came from.
- No robots.txt (a 4xx answer) means every crawler may read everything. A server error means crawlers read nothing until it is fixed.
The crawlers
What each AI crawler is for
Every description below summarizes the owner's own documentation, linked, as checked on 14 November 2026. Owners change these pages, so follow the link before you change your robots.txt.
Answer crawlers
These fetch pages for an AI search product, or when a user asks about a page. Their owners tie them to answers: block one and that product can stop showing or citing your pages. Keep them allowed on every page you want quoted.
| Crawler | What it is for | If you block it |
|---|---|---|
OAI-SearchBotOpenAI | Surfaces websites in search results in ChatGPT's search features. | OpenAI says the site is not shown in ChatGPT search answers, though it can still appear as a navigational link. |
ChatGPT-UserOpenAI | Visits pages for certain user actions in ChatGPT and custom GPTs. | OpenAI says robots.txt rules may not apply to these user-initiated visits, and that this agent does not decide whether a site appears in search. |
Claude-SearchBotAnthropic | Navigates the web to improve the quality of search results for Claude users. | Anthropic says the content is not indexed for search, which may reduce its visibility in users' search results. |
Claude-UserAnthropic | Fetches pages when people ask Claude a question. | Anthropic says Claude cannot retrieve the content in response to a user's question, which may reduce its visibility. |
PerplexityBotPerplexity | Surfaces and links websites in Perplexity's search results. Perplexity says it is not used to train foundation models. | Perplexity asks sites to allow it in order to appear in its search results. |
Perplexity-UserPerplexity | Fetches pages for user actions in Perplexity. Perplexity says it is not used for crawling or training. | Perplexity says this fetcher generally ignores robots.txt, because a user asked for the page. |
Googlebot | Google's crawler for Google Search, including all its search features. | Google names robots.txt rules for Googlebot as the control for how a site is crawled for Search, AI features included. |
Training and data crawlers
These collect pages for model training or an open dataset. Their owners' documentation doesn't tie them to answers: OpenAI says each of its settings is independent, and Apple says pages that disallow Applebot-Extended can still appear in search results. Blocking them is a policy choice about how your content is reused.
| Crawler | What it is for | If you block it |
|---|---|---|
GPTBotOpenAI | Crawls content that may be used to train OpenAI's generative AI foundation models. | Signals that the content should not be used for training. OpenAI says each of its settings is independent of the others. |
ClaudeBotAnthropic | Collects web content that may contribute to training Anthropic's models. | Anthropic says the site's future materials are left out of its model training datasets. |
Applebot-ExtendedApple | A control token, not a crawler: decides whether content Applebot crawled may train Apple's generative AI models. | Apple says pages that disallow it can still appear in search results, and it is not used for ranking. |
CCBotCommon Crawl | Common Crawl's crawler, which builds an open repository of web crawl data that others reuse. | Common Crawl documents a robots.txt group for CCBot as the way to opt out. |
Read the fine print
Google-Extended covers Gemini training and also grounding in Gemini Apps, so blocking it can keep your pages out of those answers while Google Search is unaffected. Bytespider has no current documentation from its owner; decide by your own policy.
| Crawler | What it is for | If you block it |
|---|---|---|
Google-Extended | A control token, not a separate crawler: governs whether content Google crawls may train future Gemini models and ground answers in Gemini Apps and on Vertex AI. | Google says it does not affect inclusion or ranking in Google Search. |
BytespiderByteDance | ByteDance's crawler. We found no current documentation from ByteDance on what it collects or how it is used. | Not documented by its owner. |
Deciding
Which crawlers should you block?
Decide by what each crawler is for, not by its name. Start from what you want:
| If you want to | Do this |
|---|---|
| Be named and cited in AI answers | Allow all 7 answer crawlers on the pages that answer buyers' questions. |
| Keep content out of model training | Disallow GPTBot, ClaudeBot, Applebot-Extended and CCBot; keep the answer crawlers allowed. |
| Keep content out of Gemini training only | Disallowing Google-Extended does that, but Google says it also covers grounding in Gemini Apps. Weigh that before you block it. |
| Protect private areas | Disallow the paths (/app/, /account/) in the * group, not the crawlers. |
Should you block AI crawlers? works through the trade-off in full.
Method
How the check works
- It reads one file,
/robots.txt, at the address you enter, asAmpliRank-AI-Auditor(our crawlers). It follows a redirect only within the same site; a redirect to another site is shown, not followed. - Rules are applied as the Robots Exclusion Protocol (RFC 9309, September 2022) describes: a crawler's own group, else the
*group; the longest matching rule wins, and Allow wins a tie. - It shows what robots.txt says, not what reaches your server. A firewall or bot-protection rule can still turn away a crawler that robots.txt allows, and OpenAI and Perplexity say their user-initiated fetchers may not follow robots.txt.
- The address you enter isn't stored.
Next step
Open to crawlers is step one. Being named is the measure.
The free plan runs the full AI readiness check of your site. AmpliRank then asks AI answer engines the questions your buyers ask and shows where you're named, who's named instead, and which sources decide it.
■ Ask · Measure · Compare · Trace · Fix · Share