Your robots.txt also asks Claude, GPT, Perplexity, Google not to come — which is a separate line to change from whatever your edge is doing.
robots.txt blocks four reading agents and lets search crawlers through
Your own robots.txt is what turns them away. Nothing else on the site gets a chance to be read.
robots.txt blocks four reading agents and lets search crawlers through. That is the biggest thing between you and a better grade.
5 of your 8 findings have a file below with your own URLs in it. The rest are a punch list ordered by effort, and a prompt that applies the lot one commit at a time.
Covers bbc.com. Another domain later is $5.
Your files open here in the browser the second you pay. We email a copy too, so check your spam folder for it: we are a new domain and filters have not met us yet.
Serve this at https://bbc.com/llms.txt. It tells a reading agent which of your pages matter, in the order they matter.
# BBC Home - Breaking News, World News, US News, Sports, Business, Innovation, Climate, Culture, Travel, Video & Audio > Visit BBC for trusted reporting on the latest world and US news, sports, business, climate, innovation, culture and much more. ## Updates - [BBC News - Breaking news, video and the latest top stories from the U.S. and around the world](https://www.bbc.com/news): Visit BBC News for the latest news, breaking news, video, audio and analysis. BBC News provides trusted World, U.S. and U.K. news as well as local and regional perspectives. Also entertainment, climate, business, science, technology and health news. - [US & Canada | Latest News & Updates | BBC News](https://www.bbc.com/news/us-canada): Get all the latest news, live updates and content about the US & Canada from across the BBC. - [UK | Latest News & Updates | BBC News](https://www.bbc.com/news/uk): Get all the latest news, live updates and content about the UK from across the BBC. - [Politics | Latest News & Updates | BBC News](https://www.bbc.com/news/politics): Get all the latest news, live updates and content about Politics from across the BBC. - [England | Latest News & Updates | BBC News](https://www.bbc.com/news/england): Get all the latest news, live updates and content about England from across the BBC.
Discovery is costing you the most — 15 of the 30 points you're missing. Open any line to see what we measured.
We read at most 6 pages a scan, sequentially, one second apart, and obey your robots.txt on every page rather than only the first. The full catalog and the weights are published, and blocking us takes one line.
bbc.comBBC Home - Breaking News, World News, US News, Sports, Business, Innovation, Climate, Culture, Travel, Video & AudioThis site sends a header that forbids putting it in a frame, so we are not showing one. That is a deliberate setting and a reasonable one; the name above opens it.
Visit BBC for trusted reporting on the latest world and US news, sports, business, climate, innovation, culture and much more.