Your robots.txt also asks Claude, GPT, Perplexity not to come — which is a separate line to change from whatever your edge is doing.
Four agent clients are refused at the edge
When somebody asks an assistant about you, it answers from whatever it can reach. That is not your site.
robots.txt blocks three reading agents and lets search crawlers through
Your own robots.txt is what turns them away. Nothing else on the site gets a chance to be read.
Four of the Five AI clients can't read this page at all.
6 of your 9 findings have a file below with your own URLs in it. The rest are a punch list ordered by effort, and a prompt that applies the lot one commit at a time.
Covers apnews.com. Another domain later is $5.
Your files open here in the browser the second you pay. We email a copy too, so check your spam folder for it: we are a new domain and filters have not met us yet.
Serve this at https://apnews.com/llms.txt. It tells a reading agent which of your pages matter, in the order they matter.
# Associated Press News: Breaking News, Latest Headlines and Videos | AP News > Read the latest headlines, breaking news, and videos at APNews.com, the definitive source for independent journalism from every corner of the globe.
Retrievability is costing you the most — 15 of the 43 points you're missing. Open any line to see what we measured.
We read at most 6 pages a scan, sequentially, one second apart, and obey your robots.txt on every page rather than only the first. The full catalog and the weights are published, and blocking us takes one line.
apnews.comAssociated Press News: Breaking News, Latest Headlines and Videos | AP NewsThis site sends a header that forbids putting it in a frame, so we are not showing one. That is a deliberate setting and a reasonable one; the name above opens it.
Read the latest headlines, breaking news, and videos at APNews.com, the definitive source for independent journalism from every corner of the globe.