# Six pages, one second apart, and the things we refuse to do

What our crawler requests, what it never does, and why a scanner that could get past a block would produce a worse number than one that cannot.

Published: 2026-09-09
Category: Method
Figures as of 9 September 2026, from 405 settled scans.

We measure whether sites are legible to reading agents, which means we are a crawler, which means we are the thing we would be complaining about if we did it badly. So here is exactly what ours does.

## What a scan requests

1. `GET /robots.txt`, first, always. If it disallows us, the scan ends there and we say so on the result.
2. The target page, once as each client in the catalog, sequentially, from the same address.
3. `/sitemap.xml`, `/llms.txt`, `/llms-full.txt` and four `.well-known` manifests.
4. Up to five further pages linked from the target, one second apart.

Six pages maximum, sequential, a second between them. That is a diagnostic tool, not a load generator, and it is a cap rather than a target — most scans fetch fewer.

## What it never does

- Spoof a browser user-agent to get past a block.
- Use a residential proxy or rotate addresses.
- Solve or bypass a captcha or a JavaScript challenge.
- Submit a form, sign in, or send anything that changes state.

The first one is the load-bearing one, and it is worth being precise about why. If a site refuses `BotreadyBot/1.0` and we retried as Chrome and got in, we could produce a richer report. We would also have destroyed the only thing the report is worth: the finding is that this site refuses reading agents, and a scanner that works around refusals cannot observe refusals. Our own robots.txt compliance is checked in CI, because a promise nobody tests is a sentence on a page.

> A block is the finding, not an obstacle. We record it as blocked and display it as blocked — 7% of settled scans end that way, and those scans never reach a score.

## Why the five requests go out the way they do

Same URL, same address, within a second, with only the user-agent differing. Every one of those constraints exists to remove an explanation.

- **Same URL** — otherwise you are comparing two pages.
- **Same address** — otherwise a difference could be a geographic or ASN rule rather than an agent rule.
- **Within a second** — otherwise it could be a deploy, an outage or a rate limit that reset in between.
- **Only the name changes** — so when the answers differ, the name is why.

It is a controlled comparison rather than a crawl, which is the whole reason the [status-code differential](/blog/a-403-to-claudebot-and-a-200-to-chrome) is worth anything.

## How to block us

```text
User-agent: BotreadyBot
Disallow: /
```

We read that on every scan and stop. Our full crawler statement, including the user-agent string in its exact form and where to write to us about it, is on [our crawler page](/bot).

## What we do with what we find

A scanner emits observations: status codes, byte counts, header values, character counts. Never a judgement — nothing we store says `blocked: true`. Every number a person reads is computed afterwards by a published, versioned function over those observations, which is what lets us re-score history when the method changes instead of quietly reinterpreting old numbers.

And the diagnosis is free, in full, always. Nothing about a result is blurred or held back for payment. [What we sell](/pricing) is the generated fix files, which is a different thing from the finding.

## Related

- [A 403 to ClaudeBot and a 200 to Chrome](/blog/a-403-to-claudebot-and-a-200-to-chrome): The same URL, from the same address, one second apart, answered two different ways. It is the most common serious finding we have, it is almost never deliberate, and it is usually one line to fix.
- [A user-agent string is a claim, not an identity](/blog/a-user-agent-string-is-a-claim): Every crawler dashboard in this category counts user-agent strings and calls the total AI traffic. Anyone can type ClaudeBot into a header. Here is how the identity is actually established.
- [What a median score by sector tells you, and what a global ranking does not](/blog/what-a-median-score-tells-you): The median local business scores 59. The median SaaS company scores 74. A single leaderboard across both mostly measures which kind of business a site is.