Your robots.txt also asks Claude, Perplexity not to come — which is a separate line to change from whatever your edge is doing.
Two agent clients are refused at the edge
When somebody asks an assistant about you, it answers from whatever it can reach. That is not your site.
robots.txt blocks two reading agents and lets search crawlers through
Your own robots.txt is what turns them away. Nothing else on the site gets a chance to be read.
Two of the Five AI clients can't read this page at all.
7 of your 14 findings have a file below with your own URLs in it. The rest are a punch list ordered by effort, and a prompt that applies the lot one commit at a time.
Covers theguardian.com. Another domain later is $5.
Your files open here in the browser the second you pay. We email a copy too, so check your spam folder for it: we are a new domain and filters have not met us yet.
Serve this at https://theguardian.com/llms.txt. It tells a reading agent which of your pages matter, in the order they matter.
# Latest news, sport and opinion from the Guardian > Latest US news, world news, sports, business, opinion, analysis and reviews from the Guardian, the world's leading liberal voice ## Company - [About us | The Guardian](https://www.theguardian.com/about) ## Updates - [‘This is dangerous’: slime moulds and the bitter debate over the nature of intelligence | | The Guardian](https://www.theguardian.com/news/ng-interactive/2026/sep/08/this-is-dangerous-slime-moulds-and-the-bitter-debate-over-the-nature-of-intelligence): <strong>The long read: </strong>Scientists are battling over whether supposedly simple organisms should be considered ‘intelligent’. The outcome could reshape our understanding of the natural world – and our own place within it - [Sign up for the Breaking News US email to get newsletter alerts direct to your inbox | Newsletter sign-up | The Guardian](https://www.theguardian.com/news/2026/feb/17/sign-up-for-the-breaking-news-us-email-to-get-newsletter-alerts-direct-to-your-inbox): Get the most important news as it breaks - [Back to school in Haiti and Nepal’s missing people: photos of the day – Tuesday | World news | The Guardian](https://www.theguardian.com/news/gallery/2026/sep/08/back-to-school-in-haiti-and-nepals-missing-people-photos-of-the-day-tuesday): The Guardian’s picture editors select photographs from around the world ## Other pages
Actionability is costing you the most — 15 of the 67 points you're missing. Open any line to see what we measured.
We read at most 6 pages a scan, sequentially, one second apart, and obey your robots.txt on every page rather than only the first. The full catalog and the weights are published, and blocking us takes one line.
theguardian.comLatest news, sport and opinion from the GuardianThis site sends a header that forbids putting it in a frame, so we are not showing one. That is a deliberate setting and a reasonable one; the name above opens it.
Latest US news, world news, sports, business, opinion, analysis and reviews from the Guardian, the world's leading liberal voice