Depra's AI crawler checker reads your robots.txt and your competitors' and shows, for 41 AI crawlers from 26 vendors, whether each one is allowed, partly allowed or blocked. The check is part of your Depra project, behind a login, on every plan; there is no free public checker.
robots.txt is a plain text file at yourdomain.com/robots.txt that tells bots which pages they may fetch. One wrong line in it can keep ChatGPT search, Perplexity or Google away from your pages while the site still loads normally for every human visitor.
This page covers the four kinds of AI bot, how Depra reads your file, the single URL test, how to check for llms.txt and AI bot visits from your server logs. For every bot name and what its vendor says it does, see the AI crawlers list.
Bots that learn from your site vs bots that answer questions now
The large AI companies now run separate bots for separate jobs. Depra tags each of the 41 bots with one of four bot types, because blocking each type costs you something different.
- Search bots. They index pages so an AI answer can cite them. OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, Googlebot and bingbot are in this group. Block one and you can drop out of that engine's answers. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers.
- Training bots. They collect pages to build future AI models. GPTBot, ClaudeBot and CCBot (Common Crawl) are in this group. Blocking them keeps your pages out of that training. OpenAI says its GPTBot and OAI-SearchBot settings are independent, so blocking GPTBot does not change whether you appear in ChatGPT search.
- User-triggered fetchers. They open one page when a person asks an assistant about it. ChatGPT-User and Perplexity-User are examples. They do not crawl on a schedule, and some vendors say these fetchers may not follow robots.txt.
- Control tokens. Google-Extended and Applebot-Extended are names you can write in robots.txt, but no bot visits your site under them. They tell Google and Apple how pages their normal crawlers fetched may be used, for example to train AI models.
What does blocking each bot cost you?
Blocking a search bot can remove you from that engine's answers. Blocking GPTBot keeps your content out of OpenAI model training and leaves ChatGPT search alone. Blocking Googlebot removes you from Google Search and from AI Overviews together, because AI Overviews use the normal Googlebot crawl.
The top line of the check counts two things across all 41 bots, training bots included: bots with restrictions (partly allowed or blocked) and bots that are fully open. To see only the bots that decide whether you can appear in an answer, filter the table by bot type and pick search.
Use the brand picker to switch between your own domain and any tracked competitor with a website address. The same check runs for each site, so you can see whether the brands AI names ahead of you let search bots in while your file keeps them out. Starter tracks 3 competitors, Growth 10, Scale 15 and Enterprise 20.
Check if GPTBot can crawl your site
Depra fetches the live robots.txt for each domain and gives every bot a verdict: Allowed, Partial or Blocked. The table splits the bots into two groups. Explicitly mentioned holds the bots your file names in their own User-agent line, such as User-agent: GPTBot. Following global rules holds the bots that fall back to the catch-all User-agent: * group. Click any verdict to open the robots.txt file, headed with the User-agent group that applies to that bot.
Partial means the site root is closed to that bot and an Allow line reopens part of it. If robots.txt returns a 404, no rule applies and every bot may crawl. If the file cannot be read, for example because it times out, Depra shows that error in place of verdicts.
To check by hand, open yourdomain.com/robots.txt in a browser and search for "GPTBot". If the file has a User-agent: GPTBot line, only the rules under that line apply to GPTBot. If it does not, the rules under User-agent: * apply. A Disallow: / line in the group that applies blocks the whole site. Repeat the search for OAI-SearchBot, the OpenAI bot that decides whether you can appear in ChatGPT search. The AI crawlers list has every token worth checking.
robots.txt is one layer. Your firewall, or a CDN (the service that delivers your pages, such as Cloudflare), can have its own bot setting that refuses a bot with a 403 error while robots.txt allows it. To catch that, search your server's access logs for the bot name and look at the status code on each request. Why ChatGPT doesn't mention your brand covers the other common causes.
Test one URL
Paste any page address into the URL Tester tab. Depra checks that exact address against the site's robots.txt for all 41 bots and shows Allowed or Blocked for each one, next to its bot type and platform.
Use it before you launch a new folder, after a developer edits robots.txt, or when a page you expected to see cited never shows up in AI answers.
Does your site have an llms.txt?
llms.txt is a proposed plain text file at yourdomain.com/llms.txt that points AI tools to the most useful pages on a site. Depra's crawler screen does not report it today, so check it by hand: open yourdomain.com/llms.txt in a browser. A 404 means the site has none.
It is still a proposal. No major AI search engine has said it uses the file to choose which sources to cite, so treat a missing file as a small gap and fix crawler access first. Our guide what llms.txt is and whether it works covers what the file does and how to write one.
See which AI bots visit your pages
robots.txt shows what bots may do. Your access logs show what they did. Depra reads logs from your server or CDN and reports how often AI bots reached your site, which kinds of bot they were and which pages they read.
- Formats it reads. CloudFront logs, Cloudflare and Vercel JSON log lines, JSON arrays, and Apache or nginx access logs in the standard combined format.
- How data gets in. Depra gives each log source a private web address. You set your server, CDN or a scheduled job to send log lines to it, or send it an access log file. Setup is manual, and there are no one-click plugins today.
- What is stored. Only requests from one of the 41 AI bots, matched by the name each bot sends with its request, called its user agent. Human visits are dropped before anything is saved.
- What you see. Total bot visits and the change against the previous period, bot requests over time by day, week or month, visits split into training, user query, search and other bots, and a table of the pages bots read most, which you can export as CSV. Pick a range from the last 7 days to the last 365 days.
Bots that may ignore robots.txt
Some fetchers act for one user and, by their vendor's own account, may skip robots.txt. Perplexity says Perplexity-User generally ignores robots.txt rules because a user asked for the fetch. Meta says Meta-ExternalFetcher may bypass them for the same reason.
Depra never shows Blocked for these two bots, because robots.txt cannot enforce it. Read their verdict as what your file says. The bot may still fetch pages the file closes. To stop a user-triggered fetcher, block its user agent in your firewall or CDN settings.
A scraper can copy a real bot's user agent, so the visit counts describe what your logs report.