# What we have measured

Refusal rates by client, the checks sites fail most, and the median score by kind of business — every figure an aggregate over real scans, with the count it was taken over.

Source: https://www.botready.dev/stats
Updated: 2026-09-09

Every number below is an aggregate over scans somebody asked for. None of it is modelled, sampled, extrapolated
or bought. Where a figure is missing it is because nothing has been measured yet, not because it is being held
back.

## Scans

- 409 scans have settled. 343 were scored.
- 29 (7%) refused our crawler outright and never reached a score.
- 37 ended in an error of ours or a site that could not be reached.

Refused means the site answered `BotreadyBot/1.0` with a 401, 403 or 429 on the first request and we stopped
there rather than working around it. Those scans are absent from every figure below, which means the scores
here are the scores of sites that let us read them.

## Refusal rate by client

Same URL, same address, within a second of each other.

- **Chrome** (browser control) — refused on 1% of 353 scans.
- **ClaudeBot** — refused on 9% of 353 scans.
- **GPTBot** — refused on 6% of 353 scans.
- **Perplexity** — refused on 7% of 353 scans.
- **Google-Extended** — refused on 3% of 353 scans.

Refused counts a 4xx or a 5xx. A request that produced no response at all is not counted in either direction,
because nothing was measured.

## The Google-Extended asymmetry

Of 28 sites that served a browser and refused at least one AI client, 21 served
Google-Extended anyway — 75% of them. Taken over 353 scans with a full client table.

The explanation is obvious enough — nobody wants to risk their search traffic — and it is worth noticing that
the risk is the same for all four, because none of these crawlers is the one that ranks you.

## What sites fail most

- A second page is reachable without a browser (`pages_reachable`) — failed by 100% of 1 scans.
- An agent manifest or WebMCP endpoint exists (`agent_manifest`) — failed by 90% of 359 scans.
- A markdown representation is advertised (`markdown_alternate`) — failed by 78% of 343 scans.
- A person is reachable in a way an agent can hand over (`contact_reachable`) — failed by 67% of 3 scans.
- What can be done here is declared, not just described (`action_declared`) — failed by 67% of 3 scans.
- API docs are two hops from the homepage (`api_docs_reachable`) — failed by 61% of 343 scans.
- llms.txt tells agents which pages matter (`llms_txt_present`) — failed by 42% of 359 scans.
- Last-Modified or ETag is sent (`cache_headers`) — failed by 38% of 353 scans.
- Sitemap lastmod values are maintained (`sitemap_lastmod_real`) — failed by 37% of 359 scans.
- Primary forms have labels, names and autocomplete tokens (`form_semantics`) — failed by 35% of 343 scans.

A check that could not run is counted apart from one a site failed: folding a timeout of ours into a failure
rate would blame a site for our own problem. A check a sector is exempt from leaves the denominator entirely.

## Median score by kind of site

- **general** (scoring 1.4) — median 48, range 41 to 54, from 2 sites.
- **general** (scoring 1.5) — median 64, range 64 to 64, from 1 sites.

Grouped by the profile a site was scored under, because the profile decides which checks were counted. A median
across mixed profiles averages numbers built from different denominators, which reads like a comparison and is
not one.