Six pages, one second apart, and the things we refuse to do

We measure whether sites are legible to reading agents, which means we are a crawler, which means we are the thing we would be complaining about if we did it badly. So here is exactly what ours does.

What a scan requests

1. 01GET /robots.txt, first, always. If it disallows us, the scan ends there and we say so on the result.
2. 02The target page, once as each client in the catalog, sequentially, from the same address.
3. 03/sitemap.xml, /llms.txt, /llms-full.txt and four .well-known manifests.
4. 04Up to five further pages linked from the target, one second apart.

Six pages maximum, sequential, a second between them. That is a diagnostic tool, not a load generator, and it is a cap rather than a target — most scans fetch fewer.

What it never does

• —Spoof a browser user-agent to get past a block.
• —Use a residential proxy or rotate addresses.
• —Solve or bypass a captcha or a JavaScript challenge.
• —Submit a form, sign in, or send anything that changes state.

The first one is the load-bearing one, and it is worth being precise about why. If a site refuses BotreadyBot/1.0 and we retried as Chrome and got in, we could produce a richer report. We would also have destroyed the only thing the report is worth: the finding is that this site refuses reading agents, and a scanner that works around refusals cannot observe refusals. Our own robots.txt compliance is checked in CI, because a promise nobody tests is a sentence on a page.

Why the five requests go out the way they do

Same URL, same address, within a second, with only the user-agent differing. Every one of those constraints exists to remove an explanation.

• —Same URL — otherwise you are comparing two pages.
• —Same address — otherwise a difference could be a geographic or ASN rule rather than an agent rule.
• —Within a second — otherwise it could be a deploy, an outage or a rate limit that reset in between.
• —Only the name changes — so when the answers differ, the name is why.

It is a controlled comparison rather than a crawl, which is the whole reason the status-code differential is worth anything.

How to block us

User-agent: BotreadyBot
Disallow: /

We read that on every scan and stop. Our full crawler statement, including the user-agent string in its exact form and where to write to us about it, is on our crawler page.

What we do with what we find

A scanner emits observations: status codes, byte counts, header values, character counts. Never a judgement — nothing we store says blocked: true. Every number a person reads is computed afterwards by a published, versioned function over those observations, which is what lets us re-score history when the method changes instead of quietly reinterpreting old numbers.

And the diagnosis is free, in full, always. Nothing about a result is blurred or held back for payment. What we sell is the generated fix files, which is a different thing from the finding.