A 403 to ClaudeBot and a 200 to Chrome

Nine per cent of the sites we have scanned answer 200 to a browser and 4xx to ClaudeBot. Not 200 and a thin page. Not 200 and a paywall. A refusal, from an edge that had already decided before anything of yours ran.

Nobody we have shown this to knew about it. That is the consistent part. The person who owns the site did not choose it, the person who set up the CDN did not think of it as a content decision, and the rule that produced it was probably a checkbox with a reasonable-sounding label.

Where it comes from

• —A managed bot-management ruleset with an "AI scrapers" or "AI bots" category switched to block. This is the most common single cause we see.
• —A hosting platform that turns on a bot fight mode by default and describes it as protection.
• —A rate limit written per user-agent that treats anything non-browser as abuse, at a threshold one page fetch exceeds.
• —A robots.txt Disallow: / under a specific agent name, added years ago for a scraper with a similar name.
• —A country or ASN block that happens to cover the data centre the crawler runs from.

The first three are the ones that surprise people, because they were never typed. They arrived with a default, or with a vendor's ruleset update, and the site owner's only involvement was accepting a configuration recommendation eighteen months ago.

What it looks like from the other side

Somebody asks an assistant which firm in their city does the thing you do. The assistant has your competitor's page, a directory entry about you from 2023, and a 403 where your site should be. It writes an answer. Your name may well appear in it — described by the directory entry.

Telling a block from a bad page

The distinction matters because the fixes are unrelated. We classify what came back rather than counting anything over 399 as the same event:

• —403 or 401 — an edge rule refused. Fix the rule.
• —429 — a rate limit fired. Usually a threshold, not a policy, and usually one request was enough to trip it.
• —503 with a challenge page — an interstitial. The page technically returned, and what it contains is a captcha rather than your content, which for a reading client is the same as a refusal.
• —200 with the wrong page — a soft 404, where the site says fine and hands back an error page. Rare in our corpus: one site in 350. Worth naming because it is invisible to anything that only checks status codes.
• —5xx — your origin, not a policy. A different problem and not this one.

The fix, most of the time

Allow the named agents explicitly, above whatever generic rule is catching them. On Cloudflare that is a WAF custom rule with a skip action, ordered before the managed bot ruleset. The equivalent exists on every other edge; the mechanism differs and the shape does not.

Then check that robots.txt is not separately telling them to go away, because those are two different systems and fixing one does not touch the other. Your WAF is making a decision nobody made goes through it properly.

And when you allow a crawler in, allow the real one: a user-agent string is free to type, and verifying which fetches were genuinely the agent they claimed is a separate job that most analytics gets wrong.

What we will not do to find it

If a site refuses our crawler, we record it as refused and show it as refused. We do not retry with a browser user-agent, we do not use residential proxies, and we do not solve challenges. A block is the finding — working around one would mean the number we report is a number about our evasion rather than about your site. Our crawler page states the whole limit, including how to block us.