# A user-agent string is a claim, not an identity

Every crawler dashboard in this category counts user-agent strings and calls the total AI traffic. Anyone can type ClaudeBot into a header. Here is how the identity is actually established.

Published: 2026-09-09
Category: How-to

Open your access logs, grep for `GPTBot`, count the lines. That number is being sold to you as a measure of how much AI traffic your site gets. It is a count of requests that typed a string.

```bash
curl -s https://example.com -A 'GPTBot/1.2' -o /dev/null
```

That line just added one to somebody's AI traffic. It took no permission, no infrastructure and no cleverness.

## Who is actually in your logs claiming to be a crawler

- The real crawler.
- Scrapers using a well-known name because a lot of sites allow it, which is exactly the effect the allow rule creates.
- SEO and monitoring tools checking how your site responds to crawlers — a fetch that mimics the agent on purpose.
- Us, if you scanned your own site, which is why our crawler uses its own name and never anybody else's.
- Ordinary automation with a copied header, because copying a working header is the normal way people write scripts.

## How identity is actually established

Two mechanisms, and neither of them involves trusting the string.

**Forward-confirmed reverse DNS.** Take the connecting IP, look up its PTR record, check the hostname is under the vendor's domain, then resolve that hostname forward and confirm it comes back to the same IP. Both directions matter: reverse DNS alone is asserted by whoever controls the address block.

```bash
dig +short -x 203.0.113.10
# crawl-203-0-113-10.example-vendor.com.

dig +short crawl-203-0-113-10.example-vendor.com
# 203.0.113.10   ← same address, so the claim holds
```

**A published address range.** Several vendors publish the IP ranges their crawlers fetch from as a JSON file. Check membership, refresh on a schedule, and remember that a stale copy fails honest traffic — which is worse than the problem it solves.

## Three numbers, not one

Once identity is checked rather than assumed, a fetch falls into one of three states, and collapsing them into one total throws away the interesting part:

- **Verified** — the claim was proved by rDNS or a published range. This is your actual crawler traffic.
- **Unproven** — no proof either way. Common and not sinister: a vendor with no published ranges and no PTR records leaves every fetch here.
- **Proven false** — claimed a name and came from an address that provably is not it. This is the number worth alerting on, and no dashboard that counts strings can produce it.

> We record how each fetch's identity was established alongside the fetch, and a claim with no proof is never counted as the agent it claimed to be. It is the difference between "GPTBot fetched you 400 times" and "400 requests said GPTBot, 310 of them were".

## What this does to your logs

Verification is a DNS lookup per unique address, cached — cheap enough to do on ingest rather than in a batch. The part that needs deciding up front is what you keep.

The moment you accept somebody else's traffic logs you are processing their visitors' data. Human IP addresses should be dropped or hashed in memory before anything is written down, not cleaned up on a schedule afterwards. A crawler IP needs to live only as long as the verification takes, and the verdict is the thing worth storing. We built ours that way and we would tell you to build yours that way whether or not you ever use ours.

## If you only do one thing

Do not accept a crawler-traffic number from any tool that cannot tell you how it established identity. Ask the question. The answer is usually that it grepped the user-agent.

## Related

- [Your WAF is making a content decision nobody made](/blog/your-waf-is-making-a-decision-nobody-made): A managed bot ruleset, a default left on at signup, a threshold set for scrapers. How to find the rule that is refusing reading agents, and how to let them through without letting scrapers through.
- [Six pages, one second apart, and the things we refuse to do](/blog/six-pages-one-second-apart): What our crawler requests, what it never does, and why a scanner that could get past a block would produce a worse number than one that cannot.