Your WAF is making a content decision nobody made

When a site refuses ClaudeBot, the person who owns the site almost never knows. The refusal came from a managed ruleset, or from a default that was on when they signed up, and nothing about it looked like a decision about who may read the site.

This is how to find it and how to change it. The mechanisms differ by vendor; the shape does not.

First, confirm it is the edge

Ask for the same URL twice with different names and look at the headers, not just the code:

curl -sI https://example.com -A 'Mozilla/5.0 ... Chrome/125.0 Safari/537.36'
curl -sI https://example.com -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com'

• —A cf-ray or server: cloudflare on the refusal, with none of your application headers, means Cloudflare answered and your origin never saw it.
• —A x-amzn- header or an x-cache: Error from cloudfront means AWS.
• —Your own framework headers on the 403 means your application refused, and the rule is in your code or your reverse proxy config.
• —A 503 with an HTML body containing a challenge means an interstitial. For a reading client that is a refusal with extra steps.

Cloudflare

Two separate things can be doing it, and turning off one does not touch the other.

1. 01Bot fight mode (Security → Bots). A single switch, on by default on some plans. It is blunt by design and it does not read your allow rules.
2. 02A managed bot ruleset or an AI-scraper block. Cloudflare added a one-click block for AI crawlers and a great many sites turned it on without noticing which names were in the category.
3. 03A WAF custom rule somebody wrote, often years ago, matching a user-agent substring.

To let the reading agents through, add a custom rule with a skip action and order it above the managed ruleset. Matching on user-agent alone is fine as a first move and is not proof of anything — see below.

(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "GPTBot") or
(http.user_agent contains "PerplexityBot") or
(http.user_agent contains "Google-Extended")
  → Skip: All remaining custom rules, Managed rules, Bot fight mode

AWS and everyone else

• —AWS WAF — the AWSManagedRulesBotControlRuleSet categorises crawlers. Add a scope-down statement or an explicit allow rule at a lower priority number, since AWS evaluates ascending.
• —Fastly and Akamai — bot-management products with a category for AI crawlers. Same move: an allow list evaluated before the category rule.
• —Vercel and Netlify — usually not the cause, but check any firewall or bot-protection feature you enabled.
• —nginx or Apache in front of your app — grep your configs for user_agent. A map block from 2019 blocking scrapers is a common find.

Then check robots.txt separately

These are unrelated systems. Your edge can allow a crawler that your robots.txt asks not to come, and a well-behaved crawler will obey the robots.txt and never arrive. Read your own file and look for agent-specific blocks:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

If you want them to read the site, remove those. If you do not, leave them — but then remove the edge rule too, so the decision is stated in one place instead of enforced twice by accident.

The thing to be careful about

A user-agent string is typed, not proved. An allow rule matching on the string alone will also let through anything that copies it, which is a real amount of traffic. That is an acceptable trade for getting unblocked today, and the durable version is to verify identity by reverse DNS or published IP range. A user-agent string is a claim covers how, and why nearly every crawler dashboard reports claims as visits.

Then confirm you actually fixed it

Rules cache and rulesets have ordering surprises. Re-run the two curl commands. Or run a scan and look at all five clients at once — same URL, same second, and the status codes side by side.