What we have measured so far
Every number on this page is an aggregate over scans somebody asked for. Each one carries the count it was taken over, and says what it counted. Nothing here is modelled, sampled or bought.
Refused means the site answered BotreadyBot/1.0 with a 401, 403 or 429 on the first request, and we stopped there rather than working around it. Those scans never reach a score, so they are absent from every figure below — which is worth knowing, because it means the scores on this page are the scores of sites that let us read them.
ClaudeBot is refused on 7% of the sites we have scanned. A browser asking for the same URL in the same second is refused on 0%.
Refused counts a 4xx or a 5xx. A request that produced no response at all is not counted as a refusal in either direction — nothing was measured — so the percentages are over the scans where that client got an answer. Every client is sent the same URL from the same address within a second of the others, which is what makes the comparison a comparison.
95% of the sites that block an AI client let Google’s through.
Of 151 sites that served a browser and refused at least one AI client, 143 served Google-Extended anyway. It has an obvious explanation — nobody wants to risk their search traffic — and it is worth noticing that the risk is the same for all four: none of these crawlers is the one that ranks you.
A check that could not run is counted apart from one a site failed: folding a timeout of ours into a failure rate would blame a site for our own problem. Skipped checks — the ones a sector is not measured on — leave the denominator entirely, so a rate here is over the sites the check was actually asked of.
Grouped by the profile a site was scored under, because the profile decides which checks were counted. A median across mixed profiles averages numbers built from different denominators, which reads like a comparison and is not one.
Read at 2026-10-25 00:59 UTC, from the scan record. A scan enters this page the moment it settles, so the figures move. If one of them disagrees with something we have said elsewhere, this page is the one that is current. Every check and weight is published, so a rate here can be traced to the rule that produced it.