What are GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot, OAI-SearchBot and ChatGPT-User are OpenAI’s bots for training, for ChatGPT search and for user requests. OpenAI's documentation says each robots.txt setting is independent of the others. GPTBot collects content that may be used to train OpenAI's generative AI foundation models. OAI-SearchBot finds pages for ChatGPT search, and OpenAI says sites that block it will not be shown in ChatGPT search answers.
ChatGPT-User visits a page when a person asks ChatGPT or a custom GPT about it. It does not crawl the web automatically, and OpenAI says robots.txt rules may not apply to it because a user started the request. OAI-AdsBot checks the landing pages of ads submitted to ChatGPT.
OpenAI says search changes can take about 24 hours to apply after you edit robots.txt. If you allow both GPTBot and OAI-SearchBot, OpenAI may use one crawl for both jobs.
To stay in ChatGPT search while opting out of training, add a group with the line User-agent: GPTBot followed by Disallow: /, and leave OAI-SearchBot allowed. A blocked OAI-SearchBot is one of the causes covered in why ChatGPT doesn't mention your brand.
Which Google crawler do AI Overviews use, and what are Google-Extended and GoogleOther?
Google AI Overviews and AI Mode use Googlebot, the same crawler as Google Search. An AI Overview is the AI summary at the top of some Google results. Google's guide to AI features says robots.txt rules for Googlebot are the control for Search, and a page must be indexed and eligible for a snippet to appear as a supporting link. Blocking Googlebot removes you from Search and AI Overviews together.
Google-Extended is a control token. No bot visits under that name, because Google crawls with its existing user agents. A group with User-agent: Google-Extended and Disallow: / tells Google not to use your content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. Grounding means giving Gemini pages from the Google Search index while it writes an answer. Google says Google-Extended does not affect inclusion or ranking in Google Search, so it does not remove you from AI Overviews.
Since grounding in Gemini Apps is part of what Google-Extended covers, blocking it can change how Gemini Apps use your pages at answer time. That is a different trade from blocking a bot that only collects training data, so decide on it separately. How AI answers are assembled explains the difference between grounding and training.
GoogleOther is a generic crawler that Google product teams use to fetch public pages, and Google says rules addressed to it do not affect any specific product. Google says Googlebot and GoogleOther always obey robots.txt when crawling automatically.
What are ClaudeBot, Claude-SearchBot and Claude-User?
ClaudeBot, Claude-SearchBot and Claude-User are the three bots Anthropic documents. ClaudeBot collects web content for Anthropic’s AI models. Claude-SearchBot crawls to improve search results in Claude. Claude-User fetches a page when a person asks Claude a question.
Anthropic says its bots honour robots.txt. Because each has its own name, you can block training with User-agent: ClaudeBot and Disallow: / and leave Claude-SearchBot and Claude-User allowed. The older anthropic-ai token does not appear on Anthropic's current crawler page.
What are PerplexityBot and Perplexity-User?
PerplexityBot is Perplexity’s search crawler, and Perplexity-User visits a page when a user’s question needs it. PerplexityBot finds and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models, and it follows robots.txt. Perplexity recommends allowing it in your robots.txt file.
Perplexity says that because a user asked for the fetch, Perplexity-User generally ignores robots.txt rules. A robots.txt line will not reliably stop it; a firewall or CDN rule on its user agent will.
What are Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher?
Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher are Meta’s crawlers for training, for Meta AI search and for user requests. Meta-ExternalAgent crawls for uses such as training foundation AI models and improving products. Meta-WebIndexer crawls to improve Meta AI search results. Meta-ExternalFetcher fetches single links at a user's request, and Meta says it may bypass robots.txt for that reason.
To opt out of training while staying findable in Meta AI search, disallow Meta-ExternalAgent and leave Meta-WebIndexer allowed. FacebookBot, an older Meta token, is not on Meta's current crawler page.
What are Applebot and Applebot-Extended?
Applebot is Apple’s search crawler, and Applebot-Extended is a control token, not a crawler. Applebot crawls for search in Spotlight, Siri and Safari. It follows robots.txt rules that name Applebot, and when your file names Googlebot but not Applebot, it follows the Googlebot rules.
Apple uses Applebot-Extended only to decide how pages Applebot fetched may be used. Disallowing it opts your content out of training Apple's foundation models, and Apple says those pages stay discoverable in Spotlight, Siri and Safari.
Which crawlers do Amazon, Microsoft and Common Crawl run?
Amazon lists Amazonbot, Amzn-SearchBot and Amzn-User, Microsoft crawls with bingbot, and Common Crawl uses CCBot. Amazonbot improves Amazon products and services and may be used to train Amazon AI models. Amzn-SearchBot supports search in Amazon products and is not used for model training. Amzn-User acts for users, for example on Alexa questions that need current information, and Amazon says it may not follow every robots.txt rule.
bingbot crawls for Bing. Bing says bingbot follows one group only: its own if it exists, then a msnbot group for backwards compatibility, then the catch-all User-agent: * group.
CCBot builds the free, open archive of web crawl data that Common Crawl publishes. It follows robots.txt, and Common Crawl warns that some crawlers falsely call themselves CCBot, so it suggests checking the address a request came from.
What do the Mistral, DuckDuckGo, Ai2, Cohere and other crawlers do?
Most do one of three jobs: fetch pages for AI answers, index pages for search, or collect training data. Mistral documents three agents, one for each job: MistralAI-User for pages a user asks its assistant about, MistralAI-Index for search indexing, and MistralAI-Training for training datasets. Only MistralAI-Training collects content to train Mistral's models.
DuckAssistBot fetches pages in real time for DuckDuckGo AI-assisted answers. DuckDuckGo says the data is not used to train AI models, a robots.txt change takes effect after 72 hours, and opting out does not change organic rankings on DuckDuckGo.
AI2Bot, from the Allen Institute for AI (Ai2), collects content to train open language models, and its page does not say whether it follows robots.txt. PetalBot indexes sites for Huawei's Petal Search. Diffbot follows robots.txt by default. AhrefsBot builds the index behind Ahrefs and the Yep search engine, and SemrushBot-OCOB crawls pages for Semrush's Content Toolkit.
Cohere says it runs no crawlers for model training today and lists no bot names. The cohere-ai and cohere-training-data-crawler tokens therefore have no vendor page behind them.
Which AI bots have no vendor documentation?
We found no crawler page from the company itself for TikTokSpider (ByteDance), DeepSeekBot (DeepSeek), xAI-Bot and Grok-User (xAI), Timpibot (Timpi) and Kangaroo Bot (Kangaroo LLM). Third-party bot directories describe them, but without a vendor source we make no claim about what they collect or whether they follow robots.txt.
If one of these bots puts load on your server or you do not want it, add a robots.txt group for it and also block its user agent at your firewall or CDN, since robots.txt only works for bots that choose to read it.
How do you allow AI search bots but block training bots?
Put the training bots in one group that ends with Disallow: /, then let every other bot fall through to the catch-all group. A bot follows only the group that names it, and a bot with no group of its own reads the User-agent: * rules. The example below opts out of training where the vendor offers a separate training bot or token, and keeps ChatGPT search, Claude search, Perplexity, Google and Bing able to read the site.
User-agent: GPTBotStarts a group for OpenAI's training crawlerUser-agent: ClaudeBotAdds Anthropic's training crawler to the same groupUser-agent: Meta-ExternalAgentAdds Meta's training and product crawlerUser-agent: MistralAI-TrainingAdds Mistral's training crawlerUser-agent: Applebot-ExtendedAdds Apple's training control tokenUser-agent: CCBotAdds Common Crawl's archive crawlerDisallow: /Closes the whole site to every name in this groupUser-agent: *Starts the group for every other bot, including OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and bingbotAllow: /Lets those bots read the whole site
Leave a blank line between the two groups. Google-Extended is left out on purpose because it also covers grounding in Gemini Apps; add it only if you accept that trade. If your file already has a User-agent: * group with Disallow lines for private folders, keep those lines in that group when you add Allow: /. Replacing the whole group with this example would open those folders to every bot.
How do you check which AI bots can crawl your site?
You can check any site by hand in four steps. Once you have the file open, the free robots.txt checker on this page can do steps two and three for you.
- Open the file. Go to yourdomain.com/robots.txt. A 404 means there are no rules, so every bot that follows robots.txt may crawl.
- Find each bot's group. Search the file for the token, for example OAI-SearchBot. If a User-agent line names it, only that group applies. If no line names it, the User-agent: * group applies.
- Read the rules. Disallow: / closes the whole site. A longer Allow line, such as Allow: /blog/, reopens that folder, because the most specific matching rule wins under the robots.txt standard, RFC 9309 (opens in a new tab).
- Check the server. robots.txt can allow a bot while a firewall or CDN bot setting answers it with a 403 error. Search your access logs for the bot name and look at the status codes.
Can you check robots.txt for 41 bots on your site and competitors’ sites at once?
Yes. Depra AI's AI crawler checker reads robots.txt for 41 bots on your site and on each competitor you track. Doing this by hand takes a while. The checker shows which User-agent group decides each verdict (what the AI crawler check shows), tests single URLs and reports AI bot visits from your server logs. It comes with every plan. Starter, Growth and Scale start with 7 days free, no card needed. Depra AI's own reader identifies itself as DepraBot; about DepraBot lists what it reads and the robots.txt rules it follows.
Crawler access only makes you eligible to be cited. An indexing request does not guarantee a citation either, as what indexing requests actually promise explains. To see whether AI engines then name your brand, compare generative engine optimization tools, and read how to get ChatGPT to recommend your brand for what to publish next. Terms such as grounding and AI Overview are defined in the AI search glossary.
