# Our crawler

What BotreadyBot/1.0 requests, how to block it, and the things it will never do to get past a block.

Source: https://www.botready.dev/bot
Updated: 2026-08-28

## Our user agent

```
BotreadyBot/1.0 (+https://botready.dev/bot)
```

## What we request

1. `GET /robots.txt` — first, always. If it disallows us, the scan ends there.
2. The target page, once as each client in the catalog, sequentially.
3. `/sitemap.xml`, `/llms.txt`, `/llms-full.txt` and four `.well-known` manifests.
4. Up to 5 further pages, linked from the target or named in your sitemap, 1000ms apart. Every one is checked against your robots.txt before we ask for it.

## What we never do

- Spoof a browser user agent to get past a block.
- Use a residential proxy or rotate IPs.
- Solve or bypass a captcha or a JavaScript challenge.
- Submit a form, sign in, or send anything that changes state.

A block is the finding. Working around one would destroy the only thing the number is worth.

## How to block us

```
User-agent: BotreadyBot
Disallow: /
```

We read that on every scan and stop. Questions: team@botready.dev.