A user-agent string is a claim, not an identity

Open your access logs, grep for GPTBot, count the lines. That number is being sold to you as a measure of how much AI traffic your site gets. It is a count of requests that typed a string.

curl -s https://example.com -A 'GPTBot/1.2' -o /dev/null

That line just added one to somebody's AI traffic. It took no permission, no infrastructure and no cleverness.

Who is actually in your logs claiming to be a crawler

• —The real crawler.
• —Scrapers using a well-known name because a lot of sites allow it, which is exactly the effect the allow rule creates.
• —SEO and monitoring tools checking how your site responds to crawlers — a fetch that mimics the agent on purpose.
• —Us, if you scanned your own site, which is why our crawler uses its own name and never anybody else's.
• —Ordinary automation with a copied header, because copying a working header is the normal way people write scripts.

How identity is actually established

Two mechanisms, and neither of them involves trusting the string.

Forward-confirmed reverse DNS. Take the connecting IP, look up its PTR record, check the hostname is under the vendor's domain, then resolve that hostname forward and confirm it comes back to the same IP. Both directions matter: reverse DNS alone is asserted by whoever controls the address block.

dig +short -x 203.0.113.10
# crawl-203-0-113-10.example-vendor.com.

dig +short crawl-203-0-113-10.example-vendor.com
# 203.0.113.10   ← same address, so the claim holds

A published address range. Several vendors publish the IP ranges their crawlers fetch from as a JSON file. Check membership, refresh on a schedule, and remember that a stale copy fails honest traffic — which is worse than the problem it solves.

Three numbers, not one

Once identity is checked rather than assumed, a fetch falls into one of three states, and collapsing them into one total throws away the interesting part:

• —Verified — the claim was proved by rDNS or a published range. This is your actual crawler traffic.
• —Unproven — no proof either way. Common and not sinister: a vendor with no published ranges and no PTR records leaves every fetch here.
• —Proven false — claimed a name and came from an address that provably is not it. This is the number worth alerting on, and no dashboard that counts strings can produce it.

What this does to your logs

Verification is a DNS lookup per unique address, cached — cheap enough to do on ingest rather than in a batch. The part that needs deciding up front is what you keep.

The moment you accept somebody else's traffic logs you are processing their visitors' data. Human IP addresses should be dropped or hashed in memory before anything is written down, not cleaned up on a schedule afterwards. A crawler IP needs to live only as long as the verification takes, and the verdict is the thing worth storing. We built ours that way and we would tell you to build yours that way whether or not you ever use ours.

If you only do one thing

Do not accept a crawler-traffic number from any tool that cannot tell you how it established identity. Ask the question. The answer is usually that it grepped the user-agent.