# botready.dev

> Are you BotReady? — we request your page as five different clients, compare what each one gets back, score the
> difference against a published catalog, and generate the files that close the gap.

Every page below is also available as markdown at the same URL with `.md` appended, or by sending
`Accept: text/markdown`.

## Pages

- [Are you BotReady?](https://www.botready.dev/index.md): ChatGPT, Claude and Perplexity answer questions about your category daily. See what they get from your site, and the files that fix it.
- [What we check](https://www.botready.dev/what-we-check.md): All 25 checks and every weight, published, so you can argue with the score instead of taking our word for it.
- [Pricing](https://www.botready.dev/pricing.md): The check is free and always will be. The fix pack is $15 once, and ten domains watched every week on the agency plan is $29 a month.
- [Notes from the scanner](https://www.botready.dev/blog.md): What we found measuring how legible websites are to AI agents, and how the measuring works. Ten posts, every figure taken over sites we actually scanned.
- [What we have measured](https://www.botready.dev/stats.md): Refusal rates by client, the checks sites fail most, and the median score by kind of business — every figure an aggregate over real scans, with the count it was taken over.
- [API and docs](https://www.botready.dev/docs.md): The public scan API, the fields it returns, the rate limits, and the machine-readable files this site serves.
- [Our crawler](https://www.botready.dev/bot.md): What BotreadyBot/1.0 requests, how to block it, and the things it will never do to get past a block.

## Writing

- [75% of the sites that block an AI crawler let Google's through](https://www.botready.dev/blog/the-google-extended-asymmetry.md): Of 28 sites that served a browser and refused at least one AI client, 21 served Google-Extended anyway. That is a decision about search traffic being applied to four crawlers that do not rank you.
- [A 403 to ClaudeBot and a 200 to Chrome](https://www.botready.dev/blog/a-403-to-claudebot-and-a-200-to-chrome.md): The same URL, from the same address, one second apart, answered two different ways. It is the most common serious finding we have, it is almost never deliberate, and it is usually one line to fix.
- [JavaScript is probably not your problem](https://www.botready.dev/blog/javascript-is-not-your-problem.md): Six per cent of the sites we scanned fail on client-side rendering. Nine per cent are refused at the door before rendering is even a question. The advice everyone gives is aimed at the smaller number.
- [Nine in ten sites have no agent manifest, and most of them should not](https://www.botready.dev/blog/nine-in-ten-sites-have-no-agent-manifest.md): A check that 90% of sites fail is not a finding, it is a wall. Here is why we rebuilt a whole scoring category around that, and what we changed it to.
- [Does llms.txt do anything?](https://www.botready.dev/blog/does-llms-txt-do-anything.md): We generate llms.txt files and charge for them, so read this with that in mind. The honest answer is that no major assistant is documented as fetching it, and it is still worth twenty minutes.
- [Your WAF is making a content decision nobody made](https://www.botready.dev/blog/your-waf-is-making-a-decision-nobody-made.md): A managed bot ruleset, a default left on at signup, a threshold set for scrapers. How to find the rule that is refusing reading agents, and how to let them through without letting scrapers through.
- [A user-agent string is a claim, not an identity](https://www.botready.dev/blog/a-user-agent-string-is-a-claim.md): Every crawler dashboard in this category counts user-agent strings and calls the total AI traffic. Anyone can type ClaudeBot into a header. Here is how the identity is actually established.
- [Your sitemap lastmod is probably lying](https://www.botready.dev/blog/your-sitemap-lastmod-is-probably-lying.md): Thirty-seven per cent of sites stamp every URL with the deploy timestamp. It tells a crawler nothing except when you last shipped, and it was true of this site until we checked.
- [What a median score by sector tells you, and what a global ranking does not](https://www.botready.dev/blog/what-a-median-score-tells-you.md): The median local business scores 59. The median SaaS company scores 74. A single leaderboard across both mostly measures which kind of business a site is.
- [Six pages, one second apart, and the things we refuse to do](https://www.botready.dev/blog/six-pages-one-second-apart.md): What our crawler requests, what it never does, and why a scanner that could get past a block would produce a worse number than one that cannot.

## API

- [OpenAPI description](https://www.botready.dev/openapi.json): the two public endpoints, POST /api/scan and GET /api/scan/{id}.
- [Agent manifest](https://www.botready.dev/.well-known/agent.json): what this site offers a client, in JSON.

## Optional

- [Public index](https://www.botready.dev/chart): scored sites by segment.

## Notes

- Our own crawler is `BotreadyBot/1.0 (+https://botready.dev/bot)`. It obeys robots.txt and never evades a block.
- Scans are capped at 6 pages, sequential, 1000ms apart.
