Start Free Trial (opens in a new tab)
ai crawlers & agent analytics

See which AI crawlers can read your site

Depra AI checks your robots.txt against 41 AI crawlers, with the exact rule behind each verdict, for your site and your rivals'. Test one URL, check your llms.txt, and send your server or CDN logs to see which AI bots visit and where they hit errors.

The free audit includes a short robots.txt check. No signup, no card.

Sample screen from the Depra AI app, Site / AI Crawlers: a robots.txt verdict for each AI crawler (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Bingbot) with the matching rule, counts of allowed and blocked crawlers out of 41, the llms.txt check, and a chart of AI bot visits from server logs with the page that returned errors. A test re-runs and PerplexityBot changes from Blocked to Allowed.

robots.txt verdicts

See the exact rule behind every verdict

Depra AI reads the live robots.txt and gives each of 41 AI crawlers from 26 vendors a verdict: Allowed, Partial or Blocked. Open a verdict to see the User-agent group that decides it, and switch to a rival's site to compare.
  • 41 AI crawlers from 26 vendors
  • The rule behind each verdict
  • Your rivals' files, side by side

Sample screen: a store's robots.txt next to its AI crawlers in two groups, explicitly mentioned and following global rules. As each bot is selected, the User-agent group that decides its verdict is highlighted in the file: GPTBot blocked, OAI-SearchBot allowed, PerplexityBot partial, Google-Extended blocked and ClaudeBot allowed by the catch-all group, each with a note on what the verdict costs.

url test and llms.txt

Test one page against every AI crawler

Paste any page address and Depra AI checks it against the site's robots.txt for all 41 crawlers at once, with each bot's type and platform. Depra AI also checks whether your site and your rivals' publish an llms.txt.
  • One URL, 41 crawlers at once
  • Check after every robots.txt edit
  • llms.txt, yours and your rivals'

Sample screen: the URL tester. A page address is typed and tested, and each AI crawler gets Allowed or Blocked for that address, with its bot type and platform, out of 41 crawlers. Beside it, the llms.txt check shows the file found on your site and one rival's, and missing on another's.

agent analytics

See which AI bots visit, and where they fail

robots.txt shows what bots may do. Your logs show what they did. Send your server or CDN logs to a private Depra AI address and see bot visits over time, visits by bot type, the pages bots read most and the requests that hit errors.
  • CloudFront, Cloudflare, Vercel, nginx, Apache
  • Human visits dropped before saving
  • 7 to 365 days, pages export to CSV

Sample screen: AI crawler visits to a store over the last 14 days from its server logs, the crawlers behind them (GPTBot, ClaudeBot, PerplexityBot, Google-Extended), visits split by bot type, and the pages bots read most with their status codes, including a page that returned 404 to GPTBot and one that returned 403 to OAI-SearchBot, with a CSV export.

bot types

Four kinds of AI bot, four different costs

Depra AI tags each of the 41 crawlers with its type, because blocking each type costs you something different. Filter by type to see only the bots that decide whether you can appear in an answer.

  • Search bots

    They index pages so an AI answer can cite them. Block one and you can drop out of that engine's answers.

    • OAI-SearchBot
    • PerplexityBot
    • Claude-SearchBot
    • Googlebot
    • bingbot

  • Training bots

    They collect pages to build future AI models. Blocking them keeps your pages out of that training.

    • GPTBot
    • ClaudeBot

  • User-triggered fetchers

    They open one page when a person asks an assistant about it. Some vendors say these may not follow robots.txt.

    • ChatGPT-User
    • Perplexity-User

  • Control tokens

    No bot visits under these names. They tell a vendor how it may use the pages its normal crawler fetched.

    • Google-Extended
    • Applebot-Extended
how it works

Allow or block a bot in three steps

robots.txt sits at the root of your site. Depra AI reads it and tests it; you or your developer change it.

  1. Step 01

    Find the verdict

    Open the crawler check and filter by bot type. Pick search to see the bots that decide whether you can appear in an answer.

  2. Step 02

    Give the bot its own group

    Add a User-agent line with the bot's name, then its rules. A bot with its own group stops following User-agent: *, so copy your private Disallow lines too.

  3. Step 03

    Test it again

    Publish the file, then open the check again or test a page in the URL tester to see the new verdict.

get started

Make AI answers your next sales channel

See where ChatGPT, Gemini, Perplexity, Google AI Overviews and every other major AI engine name you today, and get a ranked list of fixes to change it.

Book a DemoStart Free Trial

Join the brands and agencies using Depra AI

  • Armoray
  • Riseklix
  • Digitalyz
  • The Tooth Company
  • Kinetic Mindz
  • Drishti

Frequently Asked Questions

Answers about this part of Depra AI. Ask the team anything else.

It checks your live robots.txt against 41 AI crawlers from 26 vendors, including GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot and Claude-SearchBot from Anthropic, Googlebot and Google-Extended, PerplexityBot, Applebot, Meta-ExternalAgent, Amazonbot, bingbot and CCBot. Each one gets a verdict of Allowed, Partial or Blocked, the rule behind it, and a bot type.

It depends on what you want. Blocking GPTBot keeps your pages out of OpenAI model training, and OpenAI says it does not affect ChatGPT search, which OAI-SearchBot controls. If you want AI answers to name and cite you, keep the search bots allowed whatever you decide about training bots. Depra AI shows the verdict and the bot type for each bot; the choice stays yours.

Yes. A firewall or CDN bot setting can refuse a bot with a 403 error while robots.txt allows it, so check that bot's status codes in your access logs. It works the other way too: by their vendors' own account, Perplexity-User and Meta-ExternalFetcher may ignore robots.txt, so Depra AI never shows them as Blocked. To stop them, block their user agent in your firewall or CDN.

Depra AI gives each log source a private web address. Set your server, your CDN or a scheduled job to send log lines to it, or send it an access log file. It reads CloudFront logs, Cloudflare and Vercel JSON log lines, JSON arrays, and Apache or nginx logs in the standard combined format. Setup is manual, with no one-click plugins today. Only requests from the 41 AI bots are kept, and human visits are dropped before anything is saved.

Partly. The free AI audit includes a short robots.txt check of your site, with no signup. The full check of 41 AI crawlers, the URL test, the llms.txt check and AI bot visits from your logs come with every Depra AI plan. Starter, Growth and Scale start with 7 days free, no card needed. After the trial, Starter is ₹1,999 a month plus 18% GST.

Be the brand AI recommends

Track your brand on ChatGPT, Gemini, Perplexity, Google AI Overviews and every other major AI engine, and get a ranked list of fixes that help AI name you.

Book a DemoStart Free Trial