The AI crawler check shows which of 41 AI crawlers from 26 vendors your robots.txt file allows or blocks, with the exact rule that decides each one, for your site and your competitors' sites. robots.txt is the file on your site that tells bots which pages they may read.
The check runs inside your Depra account. There is no public, no-login version.
What does the check report for each bot?
- Allowed or blocked: with the robots.txt line that decided it.
- What the bot does: training bots collect pages to train AI models, search bots index pages so AI search can cite them, user fetchers open a page because one person asked about it, and opt-out tokens are robots.txt names, such as Google-Extended, that control AI use without being a bot.
- What blocking it costs you: a plain note on what you lose if that bot cannot read your pages.
- Advisory flag: 3 bots documented to ignore robots.txt, such as Perplexity-User, are marked advisory instead of blocked.
Can I test a single page?
Yes. Paste a URL and see, bot by bot, whether it is allowed and which rule decided it. This helps when one folder, such as /blog, has its own rules.
Does it check llms.txt?
Yes. llms.txt is a plain-text guide to your site written for AI. Depra checks whether your site and your competitors' sites publish one.
Can I see which AI bots visit my pages?
Yes, with manual setup. You send your server or CDN logs to a private Depra address, or upload a file. Depra reads CloudFront, Cloudflare, Vercel, and nginx or Apache log formats, keeps only AI bot visits and discards human traffic.
It shows which AI bots came, how often, which folders they read, and which requests got errors, such as a 403 served to OAI-SearchBot. More on the AI crawler checker.