# Notes from the scanner

What we found measuring how legible websites are to AI agents, and how the measuring works. Ten posts, every figure taken over sites we actually scanned.

Source: https://www.botready.dev/blog
Updated: 2026-09-09

## 75% of the sites that block an AI crawler let Google's through

Of 28 sites that served a browser and refused at least one AI client, 21 served Google-Extended anyway. That is a decision about search traffic being applied to four crawlers that do not rank you.

Data. Published 2026-09-09. https://www.botready.dev/blog/the-google-extended-asymmetry

## A 403 to ClaudeBot and a 200 to Chrome

The same URL, from the same address, one second apart, answered two different ways. It is the most common serious finding we have, it is almost never deliberate, and it is usually one line to fix.

Findings. Published 2026-09-09. https://www.botready.dev/blog/a-403-to-claudebot-and-a-200-to-chrome

## JavaScript is probably not your problem

Six per cent of the sites we scanned fail on client-side rendering. Nine per cent are refused at the door before rendering is even a question. The advice everyone gives is aimed at the smaller number.

Findings. Published 2026-09-09. https://www.botready.dev/blog/javascript-is-not-your-problem

## Nine in ten sites have no agent manifest, and most of them should not

A check that 90% of sites fail is not a finding, it is a wall. Here is why we rebuilt a whole scoring category around that, and what we changed it to.

Method. Published 2026-09-09. https://www.botready.dev/blog/nine-in-ten-sites-have-no-agent-manifest

## Does llms.txt do anything?

We generate llms.txt files and charge for them, so read this with that in mind. The honest answer is that no major assistant is documented as fetching it, and it is still worth twenty minutes.

Standards. Published 2026-09-09. https://www.botready.dev/blog/does-llms-txt-do-anything

## Your WAF is making a content decision nobody made

A managed bot ruleset, a default left on at signup, a threshold set for scrapers. How to find the rule that is refusing reading agents, and how to let them through without letting scrapers through.

How-to. Published 2026-09-09. https://www.botready.dev/blog/your-waf-is-making-a-decision-nobody-made

## A user-agent string is a claim, not an identity

Every crawler dashboard in this category counts user-agent strings and calls the total AI traffic. Anyone can type ClaudeBot into a header. Here is how the identity is actually established.

How-to. Published 2026-09-09. https://www.botready.dev/blog/a-user-agent-string-is-a-claim

## Your sitemap lastmod is probably lying

Thirty-seven per cent of sites stamp every URL with the deploy timestamp. It tells a crawler nothing except when you last shipped, and it was true of this site until we checked.

How-to. Published 2026-09-09. https://www.botready.dev/blog/your-sitemap-lastmod-is-probably-lying

## What a median score by sector tells you, and what a global ranking does not

The median local business scores 59. The median SaaS company scores 74. A single leaderboard across both mostly measures which kind of business a site is.

Method. Published 2026-09-09. https://www.botready.dev/blog/what-a-median-score-tells-you

## Six pages, one second apart, and the things we refuse to do

What our crawler requests, what it never does, and why a scanner that could get past a block would produce a worse number than one that cannot.

Method. Published 2026-09-09. https://www.botready.dev/blog/six-pages-one-second-apart