An AI crawler is a bot that fetches web pages for an AI company, and it names itself with a user-agent token, such as GPTBot, that you can address in robots.txt. There are three kinds: training crawlers that collect pages to build AI models, search crawlers that index pages so AI answers can cite them, and user-triggered fetchers that open a page when a person asks an assistant about it.
robots.txt is the plain text file at yourdomain.com/robots.txt that tells bots which pages they may fetch. Some vendors also publish control tokens, such as Google-Extended, which change how your content is used without any bot visiting under that name.
The table lists each bot by vendor. We checked every row against the vendor's own documentation on 15 Sep 2026, and where a vendor publishes nothing, the row says so. The sections after the table explain each vendor and give the robots.txt lines to use.
| User-agent token | Vendor | Purpose | Follows robots.txt | Source |
|---|---|---|---|---|
| GPTBot | OpenAI | Training: collects content to train OpenAI's foundation models | Yes | OpenAI docs |
| OAI-SearchBot | OpenAI | Search: finds pages to show in ChatGPT search answers | Yes | OpenAI docs |
| ChatGPT-User | OpenAI | User-triggered: visits a page when someone asks ChatGPT or a custom GPT about it | May not apply, says OpenAI | OpenAI docs |
| OAI-AdsBot | OpenAI | Checks the landing pages of ads submitted to ChatGPT; not used for training | Not stated | OpenAI docs |
| Googlebot | Search: crawls for Google Search, including AI Overviews and AI Mode | Yes | Google AI features | |
| Google-Extended | Control token, no crawler: covers Gemini model training and grounding in Gemini Apps and Vertex AI | Token only | Google crawlers | |
| GoogleOther | Generic crawler for Google product teams; its rules affect no specific product | Yes | Google crawlers | |
| ClaudeBot | Anthropic | Training: collects web content for Anthropic's AI models | Yes | Anthropic docs |
| Claude-SearchBot | Anthropic | Search: crawls to improve search results in Claude | Yes | Anthropic docs |
| Claude-User | Anthropic | User-triggered: fetches a page when someone asks Claude a question | Yes | Anthropic docs |
| anthropic-ai | Anthropic | Older token, not on Anthropic's current crawler page | Vendor docs not found | Vendor docs not found |
| PerplexityBot | Perplexity | Search: surfaces and links sites in Perplexity; not used for model training | Yes | Perplexity docs |
| Perplexity-User | Perplexity | User-triggered: visits a page to answer a user's question | Generally ignores it, says Perplexity | Perplexity docs |
| Meta-ExternalAgent | Meta | Training and products: training foundation AI models or improving products | Yes | Meta docs |
| Meta-WebIndexer | Meta | Search: improves Meta AI search results | Yes | Meta docs |
| Meta-ExternalFetcher | Meta | User-triggered: fetches single links at a user's request | May bypass it, says Meta | Meta docs |
| FacebookBot | Meta | Older token, not on Meta's current crawler page | Vendor docs not found | Vendor docs not found |
| Applebot | Apple | Search: powers search in Spotlight, Siri and Safari | Yes, and follows Googlebot rules when none name Applebot | Apple docs |
| Applebot-Extended | Apple | Control token, no crawler: opts content out of training Apple's foundation models | Token only | Apple docs |
| Amazonbot | Amazon | Improves Amazon products and services; may be used to train Amazon AI models | Yes | Amazon docs |
| Amzn-SearchBot | Amazon | Search: search in Amazon products; not used for model training | Yes | Amazon docs |
| Amzn-User | Amazon | User-triggered: for example Alexa questions that need current information | May not follow every rule, says Amazon | Amazon docs |
| bingbot | Microsoft | Search: crawls for Bing | Yes | Bing Webmaster blog |
| msnbot | Microsoft | Older Bing token; bingbot reads a msnbot group when it has no group of its own | Read by bingbot as a fallback | Bing Webmaster blog |
| CCBot | Common Crawl | Builds Common Crawl's free, open archive of web crawl data | Yes | Common Crawl |
| DuckAssistBot | DuckDuckGo | Search: fetches pages in real time for AI-assisted answers; not used for training | Yes | DuckDuckGo help |
| MistralAI-User | Mistral | User-triggered: visits a page when a user asks Mistral's assistant a question | Yes | Mistral docs |
| MistralAI-Index | Mistral | Search: crawls for indexing only; not used for training | Yes | Mistral docs |
| MistralAI-Training | Mistral | Training: builds datasets for Mistral AI models | Yes | Mistral docs |
| AI2Bot | Allen Institute for AI | Training: content used to train open language models | Not stated | Ai2 crawler page |
| AhrefsBot | Ahrefs | Search and SEO index: powers the Yep search engine and the Ahrefs platform | Yes | Ahrefs docs |
| cohere-ai | Cohere | Cohere says it runs no crawlers for model training today and lists no bot names | Vendor lists no bots | Cohere docs |
| cohere-training-data-crawler | Cohere | Not listed by Cohere, as above | Vendor lists no bots | Cohere docs |
| Diffbot | Diffbot | Crawls the web to build Diffbot's general search engine | Yes by default; can be overridden by agreement | Diffbot docs |
| PetalBot | Huawei | Search: indexes sites for Petal Search | Yes | PetalBot page |
| ImagesiftBot | Hive (ImageSift) | Collects public images, alt text and page text so ImageSift products can find similar images | Yes, and follows Googlebot rules when none name ImagesiftBot | ImageSift by Hive |
| SemrushBot-OCOB | Semrush | Crawls pages for Semrush's Content Toolkit | Yes | Semrush bots |
| Omgilibot | Webz.io | Older Webz.io crawler, replaced by Webzio and Webzio-extended | Not stated for Omgilibot | Webz.io bots |
| Bytespider | ByteDance | Vendor docs not found | Vendor docs not found | Vendor docs not found |
| TikTokSpider | ByteDance | Vendor docs not found | Vendor docs not found | Vendor docs not found |
| DeepSeekBot | DeepSeek | Vendor docs not found | Vendor docs not found | Vendor docs not found |
| Kangaroo Bot | Kangaroo LLM | Vendor docs not found | Vendor docs not found | Vendor docs not found |
| Timpibot | Timpi | Vendor docs not found | Vendor docs not found | Vendor docs not found |
| xAI-Bot | xAI | Vendor docs not found | Vendor docs not found | Vendor docs not found |
| Grok-User | xAI | Vendor docs not found | Vendor docs not found | Vendor docs not found |
| YouBot | You.com | Vendor docs not found | Vendor docs not found | Vendor docs not found |
OpenAI crawlers: GPTBot, OAI-SearchBot and ChatGPT-User
OpenAI runs separate bots for training and for search, and its documentation says each robots.txt setting is independent of the others. GPTBot collects content that may be used to train OpenAI's generative AI foundation models. OAI-SearchBot finds pages for ChatGPT search, and OpenAI says sites that block it will not be shown in ChatGPT search answers.
ChatGPT-User visits a page when a person asks ChatGPT or a custom GPT about it. It does not crawl the web automatically, and OpenAI says robots.txt rules may not apply to it because a user started the request. OAI-AdsBot checks the landing pages of ads submitted to ChatGPT.
OpenAI says search changes can take about 24 hours to apply after you edit robots.txt. If you allow both GPTBot and OAI-SearchBot, OpenAI may use one crawl for both jobs.
To stay in ChatGPT search while opting out of training, add a group with the line User-agent: GPTBot followed by Disallow: /, and leave OAI-SearchBot allowed. A blocked OAI-SearchBot is one of the causes covered in why ChatGPT doesn't mention your brand.
Google: Googlebot for AI Overviews, Google-Extended and GoogleOther
Google AI Overviews and AI Mode use Googlebot, the same crawler as Google Search. An AI Overview is the AI summary at the top of some Google results. Google's guide to AI features says robots.txt rules for Googlebot are the control for Search, and a page must be indexed and eligible for a snippet to appear as a supporting link. Blocking Googlebot removes you from Search and AI Overviews together.
Google-Extended is a control token. No bot visits under that name, because Google crawls with its existing user agents. A group with User-agent: Google-Extended and Disallow: / tells Google not to use your content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. Grounding means giving Gemini pages from the Google Search index while it writes an answer. Google says Google-Extended does not affect inclusion or ranking in Google Search, so it does not remove you from AI Overviews.
Since grounding in Gemini Apps is part of what Google-Extended covers, blocking it can change how Gemini Apps use your pages at answer time. That is a different trade from blocking a bot that only collects training data, so decide on it separately.
GoogleOther is a generic crawler that Google product teams use to fetch public pages, and Google says rules addressed to it do not affect any specific product. Google says Googlebot and GoogleOther always obey robots.txt when crawling automatically.
Anthropic crawlers: ClaudeBot, Claude-SearchBot and Claude-User
Anthropic documents three bots. ClaudeBot collects web content for its AI models. Claude-SearchBot crawls to improve search results in Claude. Claude-User fetches a page when a person asks Claude a question.
Anthropic says its bots honour robots.txt. Because each has its own name, you can block training with User-agent: ClaudeBot and Disallow: / and leave Claude-SearchBot and Claude-User allowed. The older anthropic-ai token does not appear on Anthropic's current crawler page.
Perplexity crawlers: PerplexityBot and Perplexity-User
PerplexityBot finds and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models, and it follows robots.txt. Perplexity recommends allowing it in your robots.txt file.
Perplexity-User visits a page when a user's question needs it. Perplexity says that because a user asked for the fetch, it generally ignores robots.txt rules. A robots.txt line will not reliably stop it; a firewall or CDN rule on its user agent will.
Meta crawlers: Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher
Meta-ExternalAgent crawls for uses such as training foundation AI models and improving products. Meta-WebIndexer crawls to improve Meta AI search results. Meta-ExternalFetcher fetches single links at a user's request, and Meta says it may bypass robots.txt for that reason.
To opt out of training while staying findable in Meta AI search, disallow Meta-ExternalAgent and leave Meta-WebIndexer allowed. FacebookBot, an older Meta token, is not on Meta's current crawler page.
Apple crawlers: Applebot and Applebot-Extended
Applebot crawls for search in Spotlight, Siri and Safari. It follows robots.txt rules that name Applebot, and when your file names Googlebot but not Applebot, it follows the Googlebot rules.
Applebot-Extended does not crawl. Apple uses it only to decide how pages Applebot fetched may be used. Disallowing it opts your content out of training Apple's foundation models, and Apple says those pages stay discoverable in Spotlight, Siri and Safari.
Amazon, Microsoft and Common Crawl
Amazon lists three agents. Amazonbot improves Amazon products and services and may be used to train Amazon AI models. Amzn-SearchBot supports search in Amazon products and is not used for model training. Amzn-User acts for users, for example on Alexa questions that need current information, and Amazon says it may not follow every robots.txt rule.
bingbot crawls for Bing. Bing says bingbot follows one group only: its own if it exists, then a msnbot group for backwards compatibility, then the catch-all User-agent: * group.
CCBot builds the free, open archive of web crawl data that Common Crawl publishes. It follows robots.txt, and Common Crawl warns that some crawlers falsely call themselves CCBot, so it suggests checking the address a request came from.
Other AI crawlers: Mistral, DuckDuckGo, Ai2, Cohere and more
Mistral documents three agents: MistralAI-User for pages a user asks its assistant about, MistralAI-Index for search indexing, and MistralAI-Training for training datasets. Only MistralAI-Training collects content to train Mistral's models.
DuckAssistBot fetches pages in real time for DuckDuckGo AI-assisted answers. DuckDuckGo says the data is not used to train AI models, a robots.txt change takes effect after 72 hours, and opting out does not change organic rankings on DuckDuckGo.
AI2Bot, from the Allen Institute for AI (Ai2), collects content to train open language models, and its page does not say whether it follows robots.txt. PetalBot indexes sites for Huawei's Petal Search. Diffbot follows robots.txt by default. AhrefsBot builds the index behind Ahrefs and the Yep search engine, and SemrushBot-OCOB crawls pages for Semrush's Content Toolkit.
Cohere says it runs no crawlers for model training today and lists no bot names. The cohere-ai and cohere-training-data-crawler tokens therefore have no vendor page behind them.
Which AI bots have no vendor documentation?
We found no crawler page from the company itself for Bytespider and TikTokSpider (ByteDance), DeepSeekBot (DeepSeek), xAI-Bot and Grok-User (xAI), YouBot (You.com), Timpibot (Timpi) and Kangaroo Bot (Kangaroo LLM). Third-party bot directories describe them, but without a vendor source we make no claim about what they collect or whether they follow robots.txt.
If one of these bots puts load on your server or you do not want it, add a robots.txt group for it and also block its user agent at your firewall or CDN, since robots.txt only works for bots that choose to read it.
How to allow AI search bots but block training bots
Put the training bots in one group that ends with Disallow: /, then let every other bot fall through to the catch-all group. A bot follows only the group that names it, and a bot with no group of its own reads the User-agent: * rules. The example below opts out of training where the vendor offers a separate training bot or token, and keeps ChatGPT search, Claude search, Perplexity, Google and Bing able to read the site.
| Line in robots.txt | What it does |
|---|---|
| User-agent: GPTBot | Starts a group for OpenAI's training crawler |
| User-agent: ClaudeBot | Adds Anthropic's training crawler to the same group |
| User-agent: Meta-ExternalAgent | Adds Meta's training and product crawler |
| User-agent: MistralAI-Training | Adds Mistral's training crawler |
| User-agent: Applebot-Extended | Adds Apple's training control token |
| User-agent: CCBot | Adds Common Crawl's archive crawler |
| Disallow: / | Closes the whole site to every name in this group |
| User-agent: * | Starts the group for every other bot, including OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and bingbot |
| Allow: / | Lets those bots read the whole site |
How to check which AI bots can crawl your site
You can check any site by hand in four steps.
- Open the file. Go to yourdomain.com/robots.txt. A 404 means there are no rules, so every bot that follows robots.txt may crawl.
- Find each bot's group. Search the file for the token, for example OAI-SearchBot. If a User-agent line names it, only that group applies. If no line names it, the User-agent: * group applies.
- Read the rules. Disallow: / closes the whole site. A longer Allow line, such as Allow: /blog/, reopens that folder, because the most specific matching rule wins under the robots.txt standard, RFC 9309.
- Check the server. robots.txt can allow a bot while a firewall or CDN bot setting answers it with a 403 error. Search your access logs for the bot name and look at the status codes.
Faster: check robots.txt for 41 bots on your site and your competitors' sites
Doing this by hand for 41 bots and several competitor sites takes a while. Depra's AI crawler checker reads robots.txt for your site and each competitor you track, shows which User-agent group decides each verdict, tests single URLs and reports AI bot visits from your server logs. It comes with every plan, with 7 days free and no card.
Crawler access only makes you eligible to be cited. To see whether AI engines then name your brand, compare generative engine optimization tools, and read how to get ChatGPT to recommend your brand for what to publish next. Terms such as grounding and AI Overview are defined in the AI search glossary.