# botready.dev — full text Every public page of this site, inlined, so a client needs one request rather than seven. --- # Are you BotReady? We request your page as five different clients, compare what each one gets back, and hand you the exact files that fix the gaps. Source: https://www.botready.dev/ Updated: 2026-09-04 ## What it does You give botready.dev a URL. It requests that URL as five different clients — Chrome as the control, then ClaudeBot, GPTBot, PerplexityBot and our own crawler — one second apart from the same IP, and compares what each one gets back. The headline finding is a site that answers 200 to a browser and 403 to a reading agent in the same second, which is almost always an accident nobody decided on. ## What it measures 21 checks across 6 categories, scoring version 1.2. The weights are published: - **Retrievability** — 25 of the 100. Nothing else can matter if the client cannot get the page. This carries the most weight because every other category is measured on text that was retrieved, and a refusal makes the rest unobservable rather than merely worse. - **Discovery** — 20 of the 100. The files a client asks for before it reads anything: robots.txt, a sitemap, llms.txt. Weighted high for their cost — they are the cheapest things on this list to fix and they decide what gets fetched next. - **Representation** — 20 of the 100. Whether your own account of what you do survives as data rather than as layout. An agent can only quote what it can extract. - **Structure** — 15 of the 100. The scaffolding that says which page this is and what it is about: headings, canonical URLs, JSON-LD. Below representation because a well-described page with untidy structure still gets read. - **Actionability** — 15 of the 100. Prices, plans, docs and forms an agent can act on rather than only describe. Equal to structure because being quotable and being usable are worth the same to a buyer. - **Freshness** — 5 of the 100. Dates and change signals. Worth the least because a stale signal loses a citation rarely, and it is the category where a maintained site can look bad for reasons that do not matter. ## What it costs The diagnosis is free and never blurred. The fix pack is $15 one time; monitoring is $5 per month. ## Limits we hold ourselves to - At most 6 pages per scan, sequential, 1000ms apart. - robots.txt is read first and obeyed. If it disallows us, the scan ends there. - We never spoof a user agent, use a residential proxy, or solve a captcha to get past a block. A block is a finding, not an obstacle. ## Elsewhere - [What we check](https://www.botready.dev/what-we-check): The full check catalog and the weights, published, so the score can be argued with rather than believed. - [Pricing](https://www.botready.dev/pricing): The diagnosis is free. The files are not. A one-time fix pack and monthly monitoring. - [API and docs](https://www.botready.dev/docs): The public scan API, the fields it returns, the rate limits, and the machine-readable files this site serves. - [Our crawler](https://www.botready.dev/bot): What BotreadyBot/1.0 requests, how to block it, and the things it will never do to get past a block. --- # What we check The full check catalog and the weights, published, so the score can be argued with rather than believed. Source: https://www.botready.dev/what-we-check Updated: 2026-09-02 A category is worth its weight out of 100. The checks inside it split that weight between them, and the number beside each check below is exactly what failing it takes off the score — the same number the findings list on a result prints back. A pass earns all of it, a warn earns half, a fail and an error earn none, and a check we could not run leaves the denominator rather than counting as a zero. ## The clients - `chrome` (control) — `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36` - `claudebot` (agent) — `ClaudeBot/1.0` - `gptbot` (agent) — `GPTBot/1.2` - `perplexity` (agent) — `PerplexityBot/1.0` - `googleext` (agent) — `Google-Extended` ## The checks ### Retrievability — 25 of the 100 Nothing else can matter if the client cannot get the page. This carries the most weight because every other category is measured on text that was retrieved, and a refusal makes the rest unobservable rather than merely worse. #### Every client gets the same answer (`agent_status_parity`) — 12.9 of the 100 The headline finding of the whole product: 200 to Chrome and 403 to ClaudeBot, same URL, same second. The largest single check in the catalog, because it is the only failure that makes every other measurement moot. Fails when any agent client receives a different status class than the Chrome control. #### Page text survives without JavaScript (`js_dependency_ratio`) — 7.9 of the 100 Text that appears only after JavaScript runs is text most agent clients never see. Second largest because it is the failure that looks perfect in a browser and is invisible to the people it costs you. Fails when ratio above 0.7. Warns when ratio between 0.4 and 0.7. #### First byte arrives quickly (`raw_fetch_latency`) — 2.1 of the 100 A slow first byte gets the request abandoned. Small because real timeouts are rare, and because this is the noisiest measurement in the catalog — see S2 in SCORING-PLAN.md. Fails when ttfb above 2500 ms. #### No long redirect chains (`redirect_depth`) — 2.1 of the 100 Every hop is a chance to lose the client. Small because chains are uncommon and a short one is harmless. Fails when more than 3 hops. ### Discovery — 20 of the 100 The files a client asks for before it reads anything: robots.txt, a sitemap, llms.txt. Weighted high for their cost — they are the cheapest things on this list to fix and they decide what gets fetched next. #### robots.txt exists and parses (`robots_present`) — 3.8 of the 100 Not a refusal in itself, but it is the first thing every client asks for, and a 5xx here stops some of them before they reach the page. #### robots.txt does not silently block reading agents (`robots_agent_rules`) — 7.6 of the 100 A robots.txt that disallows reading agents is a refusal you chose, usually by pasting a rule nobody read. Worth twice robots_present because this one actively turns clients away rather than merely failing to help them. Fails when an agent user agent is disallowed while search crawlers are allowed. #### Sitemap exists and is not empty (`sitemap_present`) — 3.8 of the 100 How a client finds the pages you did not link from the front door. #### llms.txt tells agents which pages matter (`llms_txt_present`) — 4.8 of the 100 The file agents look for to learn what you do in plain language. Just above the sitemap because it is the one file here we can generate for you in full, from pages we watched return 200. ### Representation — 20 of the 100 Whether your own account of what you do survives as data rather than as layout. An agent can only quote what it can extract. #### A markdown representation is advertised (`markdown_alternate`) — 7 of the 100 A markdown representation advertised in a Link header or tag is the cheapest way to hand a client clean prose instead of a layout to unpick. Fails when no link rel=alternate type=text/markdown and no Link header equivalent. #### One h1, a sane heading order, a main landmark (`semantic_landmarks`) — 6 of the 100 One h1, a sane heading order and a main landmark are how an extractor decides which part of the page is the content. #### Each page has its own title and description (`title_meta_distinct`) — 4 of the 100 Pages that share a title are pages a client cannot tell apart when it comes to cite one of them. #### Server honours Accept: text/markdown (`content_negotiation`) — 3 of the 100 Honouring Accept: text/markdown is the polite version of the markdown alternate. Small because hardly anyone implements it and its absence is unremarkable. ### Structure — 15 of the 100 The scaffolding that says which page this is and what it is about: headings, canonical URLs, JSON-LD. Below representation because a well-described page with untidy structure still gets read. #### JSON-LD describes what this is (`jsonld_present`) — 6 of the 100 On most pages this is the only place the site states what it is in a form that needs no interpretation. #### Pricing is machine readable (`pricing_structured`) — 5 of the 100 An agent quotes a price it can read as data. A price in a table is a picture of a price. Fails when prices appear only in images or JS-rendered tables. #### Canonical URL and OpenGraph are complete (`canonical_og`) — 4 of the 100 Which URL is the real one, and what the page says about itself when somebody shares it. ### Actionability — 15 of the 100 Prices, plans, docs and forms an agent can act on rather than only describe. Equal to structure because being quotable and being usable are worth the same to a buyer. #### An agent manifest or WebMCP endpoint exists (`agent_manifest`) — 5 of the 100 A manifest or WebMCP endpoint turns a page an agent can read into a service an agent can use. Scored knowing most sites will fail it for a while yet; this is the check most likely to move at the next calibration. #### API docs are two hops from the homepage (`api_docs_reachable`) — 4 of the 100 Docs two hops from the homepage are docs an agent finds while it is answering a question about you. #### Primary forms have labels, names and autocomplete tokens (`form_semantics`) — 4 of the 100 Labels, names and autocomplete tokens are what let an agent complete a form rather than describe one. #### Docs and pricing are not behind auth or a captcha (`no_wall_on_docs`) — 2 of the 100 Docs behind a login are docs no client reads. Small because it is rare, and because when it is deliberate it is a legitimate choice we should not punish hard. ### Freshness — 5 of the 100 Dates and change signals. Worth the least because a stale signal loses a citation rarely, and it is the category where a maintained site can look bad for reasons that do not matter. #### Last-Modified or ETag is sent (`cache_headers`) — 3 of the 100 Last-Modified or ETag tells a client whether reading you again is worth it. #### Sitemap lastmod values are maintained (`sitemap_lastmod_real`) — 2 of the 100 A lastmod that never changes is worse than no lastmod at all, because it is a claim rather than a gap. Fails when every lastmod is identical or in the future. ## On the weights They are our estimates. They have not yet been measured against whether a site actually gets cited, we are collecting that evidence now, and we will publish it alongside whatever it says about these numbers before we change them. Changing any of them is a versioned event: every score records the version that produced it. --- # Pricing The diagnosis is free. The files are not. A one-time fix pack and monthly monitoring. Source: https://www.botready.dev/pricing Updated: 2026-08-30 ## Free The full diagnosis. Every check, every status, every piece of evidence, and the comparison between what each client got back. Nothing is blurred and nothing is held back — you can act on all of it without paying us. ## Fix pack — $15 one time The generated files for one scan: an llms.txt built from the URLs the scan confirmed return 200, a robots.txt patch, a WAF rule that stops refusing reading agents, the JSON-LD your pages are missing, and a prompt you can hand to a coding agent to apply the rest. ## Monitoring — $5 per month Up to three claimed domains, re-scanned on a schedule, with an email the moment a client that could read you stops being able to. That regression is the one this product exists to catch, and it is silent by nature. ## Refunds If the fix pack is wrong about your site, write to us and we refund it. We would rather hear about the bad scan than keep fifteen dollars. --- # API and docs The public scan API, the fields it returns, the rate limits, and the machine-readable files this site serves. Source: https://www.botready.dev/docs Updated: 2026-09-04 ## Start a scan ```http POST https://www.botready.dev/api/scan content-type: application/json { "url": "https://example.com" } ``` Returns `{ "scanId": "...", "cached": false, "domain": "example.com" }`. A result less than 24 hours old is returned instead of crawling again, with `cached: true`. ## Read a scan ```http GET https://www.botready.dev/api/scan/{scanId} ``` Returns `status`, `scannerVersion`, `pagesCrawled`, a `progress` array of `{ key, status }` as checks land, and once `settled` is true a `score` object of `{ total, grade, scoringVersion, categoryScores, failedChecks, erroredChecks, skippedChecks }`. ## Rate limits - 5 scans an hour without an account, 50 with one. - `429` carries `retry-after` and the `x-ratelimit-*` headers. ## Machine-readable files this site serves - `/index.md` — Are you BotReady?, as markdown. - `/what-we-check.md` — What we check, as markdown. - `/pricing.md` — Pricing, as markdown. - `/docs.md` — API and docs, as markdown. - `/bot.md` — Our crawler, as markdown. - `/llms.txt` and `/llms-full.txt` - `/openapi.json` — the two endpoints above, as OpenAPI 3.1. - `/.well-known/agent.json` and `/.well-known/ai-plugin.json` - `/sitemap.xml` and `/robots.txt` Every page also answers `Accept: text/markdown` with its markdown representation, and advertises it with a `Link: <...>; rel="alternate"; type="text/markdown"` header. --- # Our crawler What BotreadyBot/1.0 requests, how to block it, and the things it will never do to get past a block. Source: https://www.botready.dev/bot Updated: 2026-08-28 ## Our user agent ``` BotreadyBot/1.0 (+https://botready.dev/bot) ``` ## What we request 1. `GET /robots.txt` — first, always. If it disallows us, the scan ends there. 2. The target page, once as each client in the catalog, sequentially. 3. `/sitemap.xml`, `/llms.txt`, `/llms-full.txt` and four `.well-known` manifests. 4. Up to 5 further pages linked from the target, 1000ms apart. ## What we never do - Spoof a browser user agent to get past a block. - Use a residential proxy or rotate IPs. - Solve or bypass a captcha or a JavaScript challenge. - Submit a form, sign in, or send anything that changes state. A block is the finding. Working around one would destroy the only thing the number is worth. ## How to block us ``` User-agent: BotreadyBot Disallow: / ``` We read that on every scan and stop. Questions: crawler@botready.dev. ---