BotreadyBot/1.0

Two lines in your robots.txt. We read it before every scan and obey it, including on the scan of your own site, and we stop at the robots.txt rather than reading anything else.

User-agent: BotreadyBot
Disallow: /

That is the whole mechanism. There is no form to fill in and no list to be removed from. If you would rather refuse us at the edge instead, a 403 to this user agent works too: we record it and stop, and the public result page for your domain says that your site refuses our scanner rather than showing a score.

A scan is one visit. In the worst case it is what follows, over about thirty seconds, sequentially, one second apart.

1. 1GET /robots.txt — first, always. If it disallows us, the scan ends here.
2. 2GET the target page as BotreadyBot. A 401, 403 or 429 ends the scan here.
3. 3GET /sitemap.xml, /llms.txt, /llms-full.txt and four /.well-known/ paths.
4. 4GET the target page five more times, once as each of Chrome, ClaudeBot, GPTBot, PerplexityBot and Google-Extended, so the responses can be compared.
5. 5One headless render of the target page, with images, fonts and media declined.
6. 6GET up to 5 more pages linked from the target, preferring /pricing and /docs.

Never more than 6 distinct pages, never concurrently, and never more than one scan of a domain in 24 hours: a second request for a domain inside that window is served the first scan’s result, so a link to your result page cannot be turned into traffic against you.

• We do not spoof a user agent to get past a block, use residential proxies, or solve captchas. If you refuse us, that is the answer, and it is the answer we publish.
• We do not submit forms, click anything, follow a login, or request a URL that carries credentials.
• We do not train anything on your content. The scan stores status codes, header values, character counts and page titles. We keep the readable text only for as long as one scan takes to compute a ratio from it.
• We do not sell your data, and there is no list to be on.

We measure how legible a site is to the clients that read it to answer questions about it. The single finding worth the whole product is a site that returns 200 to Chrome and 403 to ClaudeBot from the same address in the same second, which almost nobody chose and almost nobody knows about. Finding that requires making both requests, which is why this crawler exists and why it identifies itself.

crawler@botready.dev reaches somebody who can change the crawler’s behaviour. If we have got something wrong about your site, that is worth knowing: a check that fires on a correctly configured site is our bug.