Your robots.txt also asks Claude, Perplexity, Google not to come — which is a separate line to change from whatever your edge is doing.
Two agent clients are refused at the edge
When somebody asks an assistant about you, it answers from whatever it can reach. That is not your site.
robots.txt blocks three reading agents and lets search crawlers through
Your own robots.txt is what turns them away. Nothing else on the site gets a chance to be read.
Two of the Five AI clients can't read this page at all.
6 of your 11 findings have a file below with your own URLs in it. The rest are a punch list ordered by effort, and a prompt that applies the lot one commit at a time.
Covers theverge.com. Another domain later is $5.
Your files open here in the browser the second you pay. We email a copy too, so check your spam folder for it: we are a new domain and filters have not met us yet.
Serve this at https://theverge.com/llms.txt. It tells a reading agent which of your pages matter, in the order they matter.
# The Verge > The Verge is about technology and how it makes us feel. Founded in 2011, we offer our audience everything from breaking news to reviews to award-winning features and investigations, on our site, in video, and in podcasts. ## Company - [About The Verge | The Verge](https://www.theverge.com/about-the-verge): The Verge is about technology and how it makes us feel. Founded in 2011, we offer our audience everything from breaking news to reviews to award-winning features and investigations, on our site, in video, and in podcasts. ## Updates - [Newsletters | The Verge](https://www.theverge.com/newsletters): Enter your email and select from below to get the best of The Verge straight to your inbox. - [Another shuttered nuclear power plant is getting new life, thanks to Big Tech | The Verge](https://www.theverge.com/news/810254/google-ai-data-center-nuclear-power-plant-restart): Google made a deal to bring an Iowa nuclear plant back to life to power its data centers. ## Other pages - [Subscribe Lander | The Verge](https://www.theverge.com/subscribe): The Verge is about technology and how it makes us feel. Founded in 2011, we offer our audience everything from breaking news to reviews to award-winning features and investigations, on our site, in video, and in podcasts.
Representation is costing you the most — 19 of the 53 points you're missing. Open any line to see what we measured.
We read at most 6 pages a scan, sequentially, one second apart, and obey your robots.txt on every page rather than only the first. The full catalog and the weights are published, and blocking us takes one line.
theverge.comThe VergeThis site sends a header that forbids putting it in a frame, so we are not showing one. That is a deliberate setting and a reasonable one; the name above opens it.
The Verge is about technology and how it makes us feel. Founded in 2011, we offer our audience everything from breaking news to reviews to award-winning features and investigations, on our site, in video, and in podcasts.