About DepraBot

What does DepraBot read, and why?

Depra AI tracks how ChatGPT, Gemini, Perplexity, Google AI Overviews and every other major AI engine describe brands. Those answers cite web pages. DepraBot reads the cited pages so our customers can see the title and text of the sources behind an answer, instead of a bare URL. For a YouTube page it also asks YouTube’s oEmbed endpoint for the video title and channel name.

It also reads the homepage of a site an answer cited, about once every 90 days, to work out what kind of site it is (a publisher, a forum, a directory, a vendor). That is why you may see a request for your homepage when only a deeper page was cited.

When a customer sets up Depra AI for their own brand, it reads the homepage of the site they entered. It also checks whether that site is an online store and, if it is, reads the store’s product list, as set out in the store section below. When a customer presses “Build facts from your site”, it reads that site’s homepage and up to 5 same-site pages linked from it (such as pricing, about, shipping, returns, contact or FAQ pages), at most once a day per project.

When a customer opens their Crawlability report or tests a URL in it, it reads the robots.txt and llms.txt of their own site and of the competitor sites they entered. The feature that uses these reads is the AI crawler checker; the full list of AI bots is in the AI crawlers list.

For a customer’s Actions research, it may also read pages beyond the ones AI answers cited: a page ChatGPT read for an answer but did not cite, a list page found by a search, the brand’s own page on a marketplace or software directory (for its review count), and a contact, submission or write-for-us page on the same site as a list page it already read. These reads follow the same robots.txt rules and limits as every other DepraBot request. One customer’s research reads at most 8 to 80 pages a week, cited or not, depending on the plan, and at most 5 pages of one publication in a pass. It never reads amazon.in, nykaa.com, ajio.com, meesho.com, reddit.com or quora.com for this research.

When someone runs the free AI audit on depra.ai for a website, it reads that site’s homepage, robots.txt and llms.txt. The audit needs no account, and we do not check that the person owns the site. These reads follow the same robots.txt rules and limits as every other DepraBot request. It follows no link from the homepage. If your robots.txt keeps DepraBot out, it reads nothing past robots.txt itself, and the audit report says the homepage could not be read.

It does not crawl the web looking for pages. It follows a link from one page to another only in the cases above: a store’s sitemaps and up to 3 product pages listed in them, up to 5 pages linked from a brand’s own homepage for its facts, and one contact, submission or write-for-us page linked from a list page read for Actions research. Apart from robots.txt files, every request it makes is one of the six kinds above. The one exception is the headless browser described below, which loads a page’s own scripts and stylesheets the way any browser does.

Pages read for Actions research or Perception may also be fetched by DataForSEO, a third-party reader, under its own user agent. When DepraBot reads these pages, DataForSEO is only its backup: it fetches a page DepraBot could not read (for example blocked, timed out or too little text), only when robots.txt allows that page for both all crawlers (*) and DepraBot, and never after robots.txt refused DepraBot. Otherwise DataForSEO may read them itself, only where robots.txt allows all crawlers (*).

What does DepraBot read when a brand adds its store?

When a brand sets up a Depra AI project for its own site, DepraBot checks whether that site is an online store and, if it is and the workspace has the store read (in early access, opened per workspace), reads the store’s product list. It asks for these files, in this order, and skips the ones that do not apply:

• robots.txt, for its rules and the sitemaps it names.
• /meta.json, then /.well-known/ucp if needed, to see whether the store runs on Shopify.
• The Shopify Storefront API on the store’s own myshopify.com address, without an access token.
• /products.json, on a Shopify store, when the Storefront API did not return the full list.
• The WooCommerce Store API (/wp-json/wc/store/v1/products), on a site that is not on Shopify.
• The store’s sitemaps, when none of the APIs above lists any products.
• Up to 3 product pages listed in those sitemaps.
• The homepage, only when nothing above shows that the site sells products.

It reads only the brand’s own site, its subdomains, and the store’s own myshopify.com address, which it takes from the site’s /meta.json or /.well-known/ucp. A myshopify.com address someone types into Depra AI is asked only for its robots.txt and /meta.json unless that file names the brand’s site. A redirect to any other site is not followed. A marketplace or social site entered as a brand’s website gets no request at all.

These reads obey robots.txt like every other DepraBot request, make at most one request a second to any host, and never use the headless browser. One scan asks for at most 80 files, not counting robots.txt, starts no new request after 2 minutes, follows at most 3 redirects per file, and downloads at most 3 MB of a page, 2.5 MB of a JSON file and 10 MB of a sitemap. If your robots.txt keeps DepraBot out, a scan reads nothing past robots.txt itself, and the brand can upload its product list as a file instead.

The store read is in early access and is opened per workspace. For a workspace that has it, a scan runs when the brand sets up its project, once a week after that (scheduled between 00:30 and 02:30 UTC) unless the brand has told Depra AI it does not sell products, and when someone at the brand starts one with Fetch. Fetch is limited to 3 scans per project per day (UTC) on the Starter and Growth plans and 5 on Scale and Enterprise. Only one scan runs for a project at a time.

How can you recognise DepraBot?

The user agent on every request is DepraBot/1.0 (+https://depra.ai/about-bot). Requests come from 20.244.34.33 at the time of writing. The user agent string is the reliable identifier; the address can change.

While DepraBot is switched on, Depra AI’s API servers request nothing from your site: robots.txt and llms.txt files are read through DepraBot too. When DepraBot is switched off, the API servers read robots.txt files themselves, to check which pages a site allows crawlers to read, and, for the Crawlability report, llms.txt files. Those requests ask for nothing else and identify as DepraRobotsCheck/1.0.

How does DepraBot behave on your site?

• It obeys robots.txt as written in RFC 9309 for every page and file it asks for. It looks for a User-agent: DepraBot group first, then the * group. It honours Crawl-delay up to 10 seconds. If your robots.txt answers with a server error or does not answer at all, it treats the whole site as disallowed and comes back later; if it read your robots.txt successfully within the previous 24 hours, it keeps using that copy until then (RFC 9309 section 2.3.1.4).
• The ordinary fetch makes at most one request per second to any host and never more than two at once. It follows at most five redirects, downloads at most 2 MB of a page, and reads the HTML only: no images, media, fonts, scripts or stylesheets, no forms, no sign-in.
• For a page that serves fewer than about 200 words of text without JavaScript, a headless browser may load it as well, right after the ordinary fetch, on the first read and again whenever the page is re-read. That browser fetches the page’s scripts, stylesheets, data calls and frames the way a normal browser does, from the page’s own site and from the third-party hosts the page names, and those follow-on requests are not checked against robots.txt. Images, media and fonts are never requested, service workers are not allowed to register, and WebSocket connections are refused. Sites on our exclusion list get no request from that browser either. Such loads are capped at 500 a day across all sites.
• A page that reads fine is re-read after 7 days, then less often when it has not changed. A 403, 429 or challenge page is recorded as a block and retried later at growing intervals: a day, then three, then seven, then thirty. It uses no proxies and never disguises itself.

What does Depra AI keep from the pages DepraBot reads?

The page title, description, site name, author byline, publication date, headings, structured data types, Open Graph tags, the channel name for videos, and the page text as plain text. Only the title, the channel name for videos, the date it was last read, and the text are ever shown, and only to the customer whose tracked answers cited the page, or who has set up a Depra AI project for that domain. 180 days after the page was last cited, the page text, headings, author and Open Graph tags are deleted; the title, description, site name, channel name, dates and structured data types stay with the citation record.

A short excerpt shown in a customer's saved action (a heading, the names on a list, or one quoted line of up to 200 characters) stays with that customer's action after the page copy is deleted.

A free AI audit is stored on its own, apart from the page copies above. With it we keep the homepage’s title, description, site name, headings, structured data and a part of its text, and what robots.txt and llms.txt said. The report shows the brand name and category we drew from the homepage and the result of each check, such as whether the page has a title, one H1 and structured data, and which AI bots robots.txt allows. It does not show the page text. The report opens only from a private link. That link goes to the person who ran the audit, and a person who audits the same website within 7 days may see the same report. The Depra AI team can open it too. The audit and what DepraBot read for it are deleted after 90 days. The website name stays in our leads list.

Nothing DepraBot reads is used to train AI models, and nothing is republished.

How do you block DepraBot?

Add this to your robots.txt. DepraBot re-reads robots.txt within six hours.

User-agent: DepraBot
Disallow: /

If you cannot change robots.txt, email team@depra.ai with your domain and we will add it to the crawler’s exclusion list.

How do you contact Depra AI about DepraBot?

Questions about DepraBot or about a request you saw in your logs: team@depra.ai. Include the URL and the time if you can.