Notes from the scanner
What we found measuring how legible websites are to AI agents, and how the measuring works. Every figure in here is one we took over sites we scanned, with the date and the count beside it.
- Data3 min read
75% of the sites that block an AI crawler let Google's through
Of 28 sites that served a browser and refused at least one AI client, 21 served Google-Extended anyway. That is a decision about search traffic being applied to four crawlers that do not rank you.
· measured - Findings3 min read
A 403 to ClaudeBot and a 200 to Chrome
The same URL, from the same address, one second apart, answered two different ways. It is the most common serious finding we have, it is almost never deliberate, and it is usually one line to fix.
· measured - Findings3 min read
JavaScript is probably not your problem
Six per cent of the sites we scanned fail on client-side rendering. Nine per cent are refused at the door before rendering is even a question. The advice everyone gives is aimed at the smaller number.
· measured - Method3 min read
Nine in ten sites have no agent manifest, and most of them should not
A check that 90% of sites fail is not a finding, it is a wall. Here is why we rebuilt a whole scoring category around that, and what we changed it to.
· measured - Standards3 min read
Does llms.txt do anything?
We generate llms.txt files and charge for them, so read this with that in mind. The honest answer is that no major assistant is documented as fetching it, and it is still worth twenty minutes.
· measured - How-to3 min read
Your WAF is making a content decision nobody made
A managed bot ruleset, a default left on at signup, a threshold set for scrapers. How to find the rule that is refusing reading agents, and how to let them through without letting scrapers through.
· measured - How-to3 min read
A user-agent string is a claim, not an identity
Every crawler dashboard in this category counts user-agent strings and calls the total AI traffic. Anyone can type ClaudeBot into a header. Here is how the identity is actually established.
- How-to2 min read
Your sitemap lastmod is probably lying
Thirty-seven per cent of sites stamp every URL with the deploy timestamp. It tells a crawler nothing except when you last shipped, and it was true of this site until we checked.
· measured - Method2 min read
What a median score by sector tells you, and what a global ranking does not
The median local business scores 59. The median SaaS company scores 74. A single leaderboard across both mostly measures which kind of business a site is.
· measured - Method3 min read
Six pages, one second apart, and the things we refuse to do
What our crawler requests, what it never does, and why a scanner that could get past a block would produce a worse number than one that cannot.
· measured
Find out what your own site answers ClaudeBot.
Five clients, the same URL, one second apart. The diagnosis is free and nothing about it is blurred.