AI Crawlers: Letting Them In or Keeping Them Out
AI companies send their own bots to websites, and they do different jobs: some collect pages to train models, some build the index behind AI search answers, and some fetch a page because a person asked an assistant about it. Each company publishes the names of its bots and what blocking each one does. Blocking the wrong one can take you out of AI search results while doing nothing about training, or the reverse. This guide sets out what Google, OpenAI, Anthropic, Perplexity and Apple say about their crawlers, so you can make that choice deliberately. Checked on 5 November 2026.
The sources are each company's own documentation: Google's common crawlers list and its page on AI features and your website, OpenAI's crawler overview, Anthropic's help article on its crawlers, Perplexity's crawler page and Apple's About Applebot, plus the robots.txt standard, RFC 9309. How robots.txt works in general, and how it differs from noindex, is in our robots.txt and XML sitemaps guide.
Three kinds of AI bot
- Training crawlers collect pages that may be used to train AI models. Blocking them is about training, not visibility.
- AI search crawlers build the index that AI search features draw on and link to. Blocking them can remove your pages from those answers.
- User-triggered fetchers open a page because someone asked an assistant to. They are not crawling on their own, and some say robots.txt may not apply to them.
Most businesses that want enquiries are better off visible in AI search, and decide separately about training. If your content is your product, such as paid research or original photography, you may weigh it differently.
Google: Googlebot and Google-Extended
Google does not run a separate AI search crawler. Its AI features page says: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search." To limit what appears from your pages in AI Overviews and similar features, Google points to the snippet controls: "nosnippet, data-nosnippet, max-snippet, or noindex". Blocking Googlebot removes you from Search entirely, so that is rarely what you want. How pages get into AI Overviews and AI Mode, and how Search Console counts them, is covered in our guide to AI Overviews and your website.
Google-Extended is a separate token. Google's crawler list describes it as "a standalone product token" that controls whether content Google crawls may be used for training future Gemini models and for grounding in the Gemini Apps and Vertex AI. It "doesn't have a separate HTTP request user agent string", so you will not see it in your logs, and it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Google also notes that it intends to support a shared machine-readable AI training preference signal in early 2027.
OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User
OpenAI's overview says "each setting is independent of the others":
- OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features." Sites that block it "will not be shown in ChatGPT search answers, though can still appear as navigational links."
- GPTBot crawls content "that may be used in training our generative AI foundation models". Disallowing it "indicates a site's content should not be used in training".
- ChatGPT-User visits a page when a ChatGPT user's question calls for it. OpenAI says it "is not used for crawling the web in an automatic fashion" and that "because these actions are initiated by a user, robots.txt rules may not apply."
OpenAI notes it "can take ~24 hours" after a robots.txt change for its search systems to adjust.
Anthropic: ClaudeBot, Claude-SearchBot and Claude-User
Anthropic's help article lists three bots. ClaudeBot collects web content "that could potentially contribute to" model training; restricting it "signals that the site's future materials should be excluded from our AI model training datasets." Claude-SearchBot improves search results, and blocking it "may reduce your site's visibility and accuracy in user search results." Claude-User fetches pages when people ask Claude questions; disabling it "prevents our system from retrieving your content in response to a user query".
Anthropic says its bots honour robots.txt and the non-standard Crawl-delay rule, that the rules must be set for every subdomain, and that blocking its IP addresses instead "may not work correctly". It also says that from 30 October 2026 it has begun recognising a machine-readable "no AI training" preference based on IETF draft work, while a robots.txt rule that names ClaudeBot takes priority.
Perplexity and Apple
Perplexity lists two agents. PerplexityBot "is designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models." Perplexity-User fetches pages for users' questions, and "since a user requested the fetch, this fetcher generally ignores robots.txt rules." Changes may take up to 24 hours to apply.
Apple uses Applebot for Spotlight, Siri and Safari search, and says that data "may also be used to help train Apple foundation models". To opt out of training, disallow Applebot-Extended, which "does not crawl webpages" and only decides how Applebot's data is used; pages that disallow it "can still be included in search results." To keep a page out of AI-generated answers in Apple products, Apple points to the nosnippet robots meta tag. Apple also says that if your robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules.
An example robots.txt
A business that wants to stay visible in search and AI search but opt out of model training might use rules like these. Adjust them to your own decision.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: * Allow: /
Search and AI search crawlers (Googlebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot) fall under the last group and stay allowed. Check the result in your server logs over the following weeks. New bots appear regularly, so review the list once or twice a year.
What robots.txt cannot do
RFC 9309 describes robots.txt rules as something crawlers "are requested to honor" and states plainly: "These rules are not a form of access authorization." Its security section adds that "listing paths in the robots.txt file exposes them publicly". So:
- it does not undo past collection; Anthropic, for example, describes its opt-out as applying to the site's "future materials";
- it does not bind bots that choose to ignore it, or user-triggered fetchers that say they may not follow it;
- it does not protect private pages; put those behind a login.
Whatever you decide about bots, the content that gets cited is the content that answers questions well; our guide to writing content Google calls helpful covers that. If you are thinking about AI on your own site rather than other people's, see adding an AI chatbot to a business website.
Frequently asked questions
Does blocking Google-Extended remove my site from Google Search or AI Overviews?
No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. AI features in Search are controlled through Googlebot and the snippet controls such as nosnippet, data-nosnippet, max-snippet and noindex.
If I block GPTBot, will my site still appear in ChatGPT search?
It can. OpenAI says each setting is independent: you can allow OAI-SearchBot to appear in ChatGPT search results while disallowing GPTBot to say your content should not be used to train its models.
Do AI assistants obey robots.txt when a user asks them to open my page?
Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User because the visit is started by a user, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says disallowing Claude-User stops Claude retrieving your pages for user queries.
Is robots.txt enough to keep content private?
No. RFC 9309 says the rules are not a form of access authorization and that listing paths in robots.txt makes them publicly discoverable. Use a login or other access control for anything private.
How long do changes take to apply?
OpenAI and Perplexity both mention about 24 hours for their systems to reflect robots.txt changes. RFC 9309 lets crawlers cache robots.txt, but says they should not use a cached copy for more than 24 hours unless the file cannot be reached.
Want us to review your robots.txt and crawler settings? See our website maintenance service and all our services, or email [email protected].