About DepraBot

Depra AI tracks how ChatGPT, Gemini, Perplexity and Google AI Overviews describe brands. Those answers cite web pages. DepraBot reads the cited pages so our customers can see the title and text of the sources behind an answer, instead of a bare URL. For a YouTube page it also asks YouTube’s oEmbed endpoint for the video title and channel name.

It also reads the homepage of a site an answer cited, about once every 90 days, to work out what kind of site it is (a publisher, a forum, a directory, a vendor). That is why you may see a request for your homepage when only a deeper page was cited.

When a customer sets up Depra for their own brand, it reads up to eight standard pages of the site they entered: the homepage, /about, /about-us, /pricing, /products, /services, /solutions and /features.

When a customer opens their Crawlability report or tests a URL in it, it reads the robots.txt and llms.txt of their own site and of the competitor sites they entered.

It does not crawl the web looking for pages and it does not follow links from one page to the next. Every page it asks for is one of the four above. The one exception is the headless browser described below, which loads a page’s own scripts and stylesheets the way any browser does.

The user agent on every request is DepraBot/1.0 (+https://depra.ai/about-bot). Requests come from 4.240.90.125 at the time of writing. The user agent string is the reliable identifier; the address can change.

• It obeys robots.txt as written in RFC 9309 for every page it asks for. It looks for a User-agent: DepraBot group first, then the * group. It honours Crawl-delay up to 10 seconds. If your robots.txt answers with a server error or does not answer at all, it treats the whole site as disallowed and comes back later; if it read your robots.txt successfully within the previous 24 hours, it keeps using that copy until then (RFC 9309 section 2.3.1.4).
• The ordinary fetch makes at most one request per second to any site and never more than two at once. It follows at most five redirects, downloads at most 2 MB of a page, and reads the HTML only: no images, media, fonts, scripts or stylesheets, no forms, no sign-in.
• For a page that serves fewer than about 200 words of text without JavaScript, a headless browser may load it as well, right after the ordinary fetch, on the first read and again whenever the page is re-read. That browser fetches the page’s scripts, stylesheets, data calls and frames the way a normal browser does, from the page’s own site and from the third-party hosts the page names, and those follow-on requests are not checked against robots.txt. Images, media and fonts are never requested, service workers are not allowed to register, and WebSocket connections are refused. Sites on our exclusion list get no request from that browser either. Such loads are capped at 500 a day across all sites.
• A page that reads fine is re-read after 7 days, then less often when it has not changed. A 403, 429 or challenge page is recorded as a block and retried later at growing intervals: a day, then three, then seven, then thirty. It uses no proxies and never disguises itself.

The page title, description, site name, author byline, publication date, headings, structured data types, Open Graph tags, the channel name for videos, and the page text as plain text. Only the title, the channel name for videos, the date it was last read, and the text are ever shown, and only to the customer whose tracked answers cited the page, or who has set up a Depra project for that domain. 180 days after the page was last cited, the page text, headings, author and Open Graph tags are deleted; the title, description, site name, channel name, dates and structured data types stay with the citation record.

Nothing DepraBot reads is used to train AI models, and nothing is republished.

Add this to your robots.txt. DepraBot re-reads robots.txt within six hours.

User-agent: DepraBot
Disallow: /

If you cannot change robots.txt, email team@depra.ai with your domain and we will add it to the crawler’s exclusion list.

Questions about DepraBot or about a request you saw in your logs: team@depra.ai. Include the URL and the time if you can.