Short answer
The AI crawler check shows which of 41 AI crawlers from 26 vendors your robots.txt file allows or blocks, with the exact rule that decides each one, for your site and your competitors' sites. robots.txt is the file on your site that tells bots which pages they may read.
The full check of 41 AI crawlers runs inside your Depra AI account. The free AI audit includes a short robots.txt check of your site, with no signup.
What does the check report for each bot?
- Allowed or blocked: with the robots.txt line that decided it.
- Bot type: one of the four types in the table below.
- What blocking it costs you: a plain note on what you lose if that bot cannot read your pages.
- Advisory flag: 3 bots documented to ignore robots.txt, such as Perplexity-User, are marked advisory instead of blocked.
| Bot type | What it does |
|---|---|
| Training bot | Collects pages to train AI models |
| Search bot | Indexes pages so AI search can cite them |
| User fetcher | Opens a page because one person asked about it |
| Opt-out token | A robots.txt name, such as Google-Extended, that controls AI use without being a bot |
Can I test a single page?
Yes. Paste a URL and see, bot by bot, whether it is allowed and which rule decided it. This helps when one folder, such as /blog, has its own rules.
Does it check llms.txt?
Yes. llms.txt is a plain-text guide to your site written for AI. Depra AI checks whether your site and your competitors' sites publish one. Our post on whether indexing requests get you into AI answers also explains what llms.txt is for.
Can I see which AI bots visit my pages?
Yes, with manual setup. You send your server or CDN logs to a private Depra AI address, or upload a file. Depra AI reads CloudFront, Cloudflare, Vercel, and nginx or Apache log formats, keeps only AI bot visits and discards human traffic.
It shows which AI bots came, how often, which folders they read, and which requests got errors, such as a 403 served to OAI-SearchBot. More on the AI crawler checker.
FAQ
How do I check if GPTBot can crawl my site?
Open yoursite.com/robots.txt and look for a 'User-agent: GPTBot' group, or a 'User-agent: *' group, with Disallow lines. GPTBot is OpenAI's training crawler, and OAI-SearchBot fetches pages for ChatGPT search answers, so check both. Depra AI reads these rules for 41 AI crawlers and shows the line that decides each one.
Does blocking GPTBot stop ChatGPT from citing my site?
Not on its own. OpenAI lists GPTBot and OAI-SearchBot as separate bots with separate robots.txt rules, so blocking GPTBot leaves OAI-SearchBot's access unchanged. Source: OpenAI's crawler documentation (opens in a new tab), checked 7 Oct 2026.
Check which AI bots can read your site
Free AI audit: 5 English questions on ChatGPT, Gemini, Perplexity and Google AI Overviews, once. No signup, no card.
