Skip to content

Articles / How engines answer

Should you block AI crawlers? What each one does, and what blocking costs

Block training crawlers if you object to training; keep the crawlers that fetch pages for AI answers allowed, or those engines can stop citing you.

Published
Reading time
7 min read
By
AmpliRank
Tags
AI Crawlers · robots.txt · AI Search

Block AI training crawlers if you don’t want your content used to train models, but keep the crawlers that fetch pages for AI answers allowed, or those products can stop showing and citing your pages. The companies that run the crawlers document which is which, and their documentation, not the crawler’s name, is the guide to follow.

In short

  • OpenAI, Anthropic and Perplexity each run separate crawlers for search, for user requests and, for OpenAI and Anthropic, for training. Each one obeys its own robots.txt group.
  • Blocking a search or user crawler can take you out of that product’s answers. OpenAI says a site that opts out of OAI-SearchBot is not shown in ChatGPT search answers.
  • Blocking a training crawler is a policy choice. OpenAI says its settings are independent, and Apple says pages that disallow Applebot-Extended can still appear in search results.
  • Google-Extended is the exception to watch: Google says it covers grounding in Gemini Apps as well as training, while Google Search is unaffected.
  • robots.txt is a request, not a lock. Some user-initiated fetchers may not follow it, and a firewall can block a crawler robots.txt allows.

Why is “block AI crawlers” the wrong question?

“AI crawlers” is not one thing. Each company runs several crawlers with different jobs, and robots.txt rules apply to one user-agent token at a time. A rule written to stop training can also stop the crawler that lets an engine cite you, if it is aimed at the wrong token or at every crawler at once.

The Robots Exclusion Protocol, published as RFC 9309 in September 2022, sets how crawlers read the file: a crawler follows the group that names it, and falls back to the User-agent: * group only when none does. So User-agent: * with Disallow: / blocks every crawler without its own group, the answer crawlers included.

Which crawlers fetch the pages an engine reads matters because of query fan-out: engines search the web while answering, and they can only cite pages they were allowed to fetch.

What does each AI crawler do?

Each crawler does what its owner’s documentation says, and the documents sort most of them into three groups: crawlers that serve answers, crawlers that collect training data, and control tokens. The table summarizes each owner’s page as of November 2026; follow the links in Sources before changing a rule, because owners revise these pages.

Crawler Owner What the owner says it is for Tied to answers?
OAI-SearchBot OpenAI Surfaces websites in ChatGPT’s search features Yes
ChatGPT-User OpenAI Visits pages for user actions in ChatGPT and custom GPTs Yes, user-initiated
GPTBot OpenAI Crawls content that may train OpenAI’s foundation models No
Claude-SearchBot Anthropic Improves search result quality for Claude users Yes
Claude-User Anthropic Fetches pages when people ask Claude a question Yes, user-initiated
ClaudeBot Anthropic Collects content that may train Anthropic’s models No
PerplexityBot Perplexity Surfaces and links websites in Perplexity’s search results Yes
Perplexity-User Perplexity Fetches pages for user actions in Perplexity Yes, user-initiated
Googlebot Google Crawls for Google Search and all its features Yes
Google-Extended Google Control token for Gemini training and grounding in Gemini Apps Partly
Applebot-Extended Apple Control token for training Apple’s generative models No
CCBot Common Crawl Builds an open repository of web crawl data No
Bytespider ByteDance No current owner documentation found Unknown

Two of these are not crawlers at all. Google says Google-Extended “doesn’t have a separate HTTP request user agent string”, and Apple says Applebot-Extended does not crawl webpages. Both are tokens that tell the company how it may use pages its main crawler already fetched.

What do you give up by blocking each kind?

What you give up depends on the kind of crawler, and the owners spell it out:

  • Search crawlers. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says blocking Claude-SearchBot may reduce your visibility in its users’ search results. Perplexity asks site owners to allow PerplexityBot to appear in its search results.
  • User-initiated fetchers. Anthropic says blocking Claude-User stops Claude retrieving your content in response to a user’s question. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.
  • Googlebot. Google’s guidance on AI features names robots.txt rules for Googlebot as the control for how a site is crawled for Search, and says a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode. Blocking Googlebot removes you from all of it.
  • Training crawlers. OpenAI says each of its settings is independent of the others, so blocking GPTBot does not opt you out of ChatGPT search. Anthropic describes blocking ClaudeBot as excluding your future content from training. Apple says pages that disallow Applebot-Extended can still be included in search results.
  • Google-Extended. Google says it does not affect inclusion or ranking in Google Search. It does cover grounding in Gemini Apps and Grounding with Google Search on Vertex AI, so blocking it can keep your pages out of Gemini answers that would otherwise draw on them.

The asymmetry is the point. Blocking a training crawler has no documented cost in answers; blocking an answer crawler has a documented one.

How do you decide which to block?

Decide by goal, then write one group per token. This table covers the common goals:

Your goal Allow Disallow Watch out for
Be named and cited in AI answers OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot Nothing A * group that disallows /
Opt out of model training, stay in answers All answer crawlers GPTBot, ClaudeBot, Applebot-Extended, CCBot Google-Extended also covers Gemini grounding
Opt out of Gemini training as well All answer crawlers Add Google-Extended Gemini Apps may stop grounding answers on your pages
Keep private areas private All crawlers on public pages Paths such as /app/ in the * group robots.txt is public and not access control: put private pages behind a sign-in

For a B2B company that wants buyers to find it through AI answers, the first two rows fit: every answer crawler allowed on the pages that answer buyers’ questions, and training crawlers allowed or blocked as a matter of policy.

Check your own file. The free AI crawler check reads your robots.txt and shows, for each of these 13 crawlers, whether it may read your home page and which rule decided it.

What can go wrong?

A robots.txt change can do more, or less, than you meant. The common failure modes:

  1. The wildcard catches everyone. A User-agent: * group that disallows / blocks every answer crawler that has no group of its own. Give each crawler you want its own group, or allow / in the wildcard and disallow only private paths.
  2. A server error blocks everything. RFC 9309 says a crawler that gets a server error for robots.txt must assume it may crawl nothing. A missing file (a 4xx) means the opposite: everything is allowed.
  3. The file allows it, the firewall doesn’t. Bot protection can turn away a crawler that robots.txt allows. The two settings live in different places, so check both.
  4. Changes take time. OpenAI and Perplexity both say a robots.txt change can take about 24 hours to reach their systems.
  5. An undocumented crawler. We found no current documentation from ByteDance on Bytespider. With no owner statement to weigh, decide by your own policy.

Blocking or unblocking a crawler changes what an engine can read, not whether it names you. Whether you are named is a sampled estimate: measure the same questions before and after the change, report the share of answers that name you with its range, and call any change observed, not caused.

Where to start

Run your domain through the AI crawler check and look first at the answer crawlers: any of the seven blocked is worth fixing this week, unless you blocked it on purpose. Then decide the training crawlers as a policy question, write one group per token, and see on a sample run which sources the engines cite when they answer your buyers’ questions.

Sources

■ Ask · Measure · Compare · Trace · Fix · Share

Measure what AI says about you.

Cookies