Data

75% of the sites that block an AI crawler let Google's through

Of 28 sites that served a browser and refused at least one AI client, 21 served Google-Extended anyway. That is a decision about search traffic being applied to four crawlers that do not rank you.

9 September 2026·3 min read

Figures as of 9 September 2026, taken over 350 scanned sites. They move as scans arrive — the live version is here.

We request every site as five clients: Chrome as the control, then ClaudeBot, GPTBot, PerplexityBot and Google-Extended. Same URL, same address, within a second of each other. The only thing that changes between the five requests is the user-agent string, which makes the comparison a comparison.

Across 350 sites, 345 served a browser. Of those, 28 refused at least one AI client. And 21 of those 28 — three quarters — served Google-Extended anyway.

of sites refused ClaudeBot

refused PerplexityBot

refused GPTBot

refused Google-Extended

Why the shape is what it is

The explanation is not mysterious. Somebody looked at a bot-management rule, recognised one of the names in it, and did not want to find out what happens to their search traffic. Everything with Google in the name got an exception. Nothing else did.

The thing worth noticing is that the exception does not do what it was made to do. Google-Extended is not the crawler that ranks you. It has no effect on Search indexing or on ranking at all — it is the control for Gemini and for grounding in AI features. Googlebot is the one that ranks you, and it is a different token that most bot-management rules already allow.

What a refusal costs

An assistant that cannot fetch your page does not report an error to the person who asked. It answers anyway, from whatever it has: a directory listing, a review site, a competitor who wrote about you, a cached description from two years ago. You are not absent from the answer. You are described by somebody else in it.

That is the difference between this and a search-engine problem. A page missing from an index is missing. A page missing from a model's reach is replaced.

How to check yours in a minute

You do not need us for this. From a terminal, ask for your own homepage twice with different names on it:

curl -s -o /dev/null -w '%{http_code}\n' https://example.com \
  -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0 Safari/537.36'

curl -s -o /dev/null -w '%{http_code}\n' https://example.com \
  -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com'

Two 200s and there is nothing here for you. A 200 and a 403 and you have found the thing. Repeat with GPTBot/1.2 and PerplexityBot/1.0 and Google-Extended, because they are frequently not treated the same.

If you would rather see all five side by side with the headers and the timing, run a scan — it is free, there is no account, and nothing about the result is blurred.

On the number itself

Twenty-one of twenty-eight is a small denominator and we would rather say so than round it into a headline. It is 75% of the sites that refused somebody, not 75% of the internet, and the population it describes is sites people asked us to scan — which skews toward sites somebody already suspected had a problem.

The live version of this figure, and the count it is currently taken over, is on what we have measured. It moves every time a scan settles.