# Your WAF is making a content decision nobody made

A managed bot ruleset, a default left on at signup, a threshold set for scrapers. How to find the rule that is refusing reading agents, and how to let them through without letting scrapers through.

Published: 2026-09-09
Category: How-to
Figures as of 9 September 2026, from 350 scanned sites.

When a site refuses ClaudeBot, the person who owns the site almost never knows. The refusal came from a managed ruleset, or from a default that was on when they signed up, and nothing about it looked like a decision about who may read the site.

This is how to find it and how to change it. The mechanisms differ by vendor; the shape does not.

## First, confirm it is the edge

Ask for the same URL twice with different names and look at the headers, not just the code:

```bash
curl -sI https://example.com -A 'Mozilla/5.0 ... Chrome/125.0 Safari/537.36'
curl -sI https://example.com -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com'
```

- A `cf-ray` or `server: cloudflare` on the refusal, with none of your application headers, means Cloudflare answered and your origin never saw it.
- A `x-amzn-` header or an `x-cache: Error from cloudfront` means AWS.
- Your own framework headers on the 403 means your application refused, and the rule is in your code or your reverse proxy config.
- A 503 with an HTML body containing a challenge means an interstitial. For a reading client that is a refusal with extra steps.

## Cloudflare

Two separate things can be doing it, and turning off one does not touch the other.

1. **Bot fight mode** (Security → Bots). A single switch, on by default on some plans. It is blunt by design and it does not read your allow rules.
2. **A managed bot ruleset or an AI-scraper block.** Cloudflare added a one-click block for AI crawlers and a great many sites turned it on without noticing which names were in the category.
3. **A WAF custom rule** somebody wrote, often years ago, matching a user-agent substring.

To let the reading agents through, add a custom rule with a **skip** action and order it above the managed ruleset. Matching on user-agent alone is fine as a first move and is not proof of anything — see below.

```text
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "GPTBot") or
(http.user_agent contains "PerplexityBot") or
(http.user_agent contains "Google-Extended")
  → Skip: All remaining custom rules, Managed rules, Bot fight mode
```

## AWS and everyone else

- **AWS WAF** — the `AWSManagedRulesBotControlRuleSet` categorises crawlers. Add a scope-down statement or an explicit allow rule at a lower priority number, since AWS evaluates ascending.
- **Fastly and Akamai** — bot-management products with a category for AI crawlers. Same move: an allow list evaluated before the category rule.
- **Vercel and Netlify** — usually not the cause, but check any firewall or bot-protection feature you enabled.
- **nginx or Apache in front of your app** — grep your configs for `user_agent`. A `map` block from 2019 blocking scrapers is a common find.

## Then check robots.txt separately

These are unrelated systems. Your edge can allow a crawler that your robots.txt asks not to come, and a well-behaved crawler will obey the robots.txt and never arrive. Read your own file and look for agent-specific blocks:

```text
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /
```

If you want them to read the site, remove those. If you do not, leave them — but then remove the edge rule too, so the decision is stated in one place instead of enforced twice by accident.

## The thing to be careful about

A user-agent string is typed, not proved. An allow rule matching on the string alone will also let through anything that copies it, which is a real amount of traffic. That is an acceptable trade for getting unblocked today, and the durable version is to verify identity by reverse DNS or published IP range. [A user-agent string is a claim](/blog/a-user-agent-string-is-a-claim) covers how, and why nearly every crawler dashboard reports claims as visits.

## Then confirm you actually fixed it

Rules cache and rulesets have ordering surprises. Re-run the two curl commands. Or [run a scan](/scan) and look at all five clients at once — same URL, same second, and the status codes side by side.

## Related

- [A 403 to ClaudeBot and a 200 to Chrome](/blog/a-403-to-claudebot-and-a-200-to-chrome): The same URL, from the same address, one second apart, answered two different ways. It is the most common serious finding we have, it is almost never deliberate, and it is usually one line to fix.
- [A user-agent string is a claim, not an identity](/blog/a-user-agent-string-is-a-claim): Every crawler dashboard in this category counts user-agent strings and calls the total AI traffic. Anyone can type ClaudeBot into a header. Here is how the identity is actually established.
- [75% of the sites that block an AI crawler let Google's through](/blog/the-google-extended-asymmetry): Of 28 sites that served a browser and refused at least one AI client, 21 served Google-Extended anyway. That is a decision about search traffic being applied to four crawlers that do not rank you.