seobot.dk
💎 Pricing📘 SEO Guides🤖 llms.txt Gen🧠 Deep Dives📖 Blog
Sign In
Back to Insights
Ahrefs

We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read

Are llms.txt Files Worth It? Ahrefs Analyzes 137K Sites to Find Out

In the race to prepare for the "AI-driven web," a new standard has emerged: the llms.txt file. Designed as a machine-readable version of your website's key information, it's marketed as the ultimate way to ensure Large Language Models (LLMs) understand your content correctly. But does it actually work, or is it just another piece of digital clutter?

A recent massive study by Ahrefs provides a sobering answer. By analyzing server logs and live traffic from 137,000 domains, the data reveals a startling reality: 97% of llms.txt files are never even read by bots.

What is llms.txt and Why Was It Created?

The llms.txt file is a proposed standard (similar to robots.txt) located in the root directory of a website. Its primary goal is to provide a concise, markdown-formatted summary of a website's purpose and a directory of its most important pages.

Since LLMs often struggle with complex HTML structures or get lost in navigation menus, llms.txt serves as a "cheat sheet" for AI crawlers to quickly ingest the most relevant data without wasting tokens or computing power.

The Data: A Massive Gap Between Intent and Reality

Ahrefs utilized their Web Analytics and Bot Analytics tools to track every single user agent hitting these 137K domains. The findings are clear:

  • Negligible Adoption: Despite the buzz in the AI community, the vast majority of AI crawlers are not looking for this file.
  • The 3% Minority: Only a tiny fraction of sites saw their llms.txt files accessed, suggesting that most current LLM crawlers still rely on traditional scraping and indexing methods.
  • The Google Factor: The study notes that Google's approach to AI indexing remains distinct and doesn't currently prioritize this specific file format for its primary crawling mechanisms.

Why This Matters for Your SEO Strategy

As an SEO professional or webmaster, your time is your most valuable asset. This data tells us that prioritizing llms.txt over fundamental SEO is a mistake.

While the concept of "AI-readiness" is important, the current infrastructure of the web still relies on structured data (Schema.org), high-quality content, and clean HTML. If you are spending hours crafting a perfect llms.txt while your Core Web Vitals are failing or your metadata is missing, you are optimizing for a ghost.

However, this doesn't mean you should ignore AI optimization entirely. It simply means the method of delivery is still evolving. The goal remains the same: making your data easy for any machine—be it Googlebot or GPT-Bot—to parse.

Final Verdict: Should You Implement It?

If it takes you five minutes to set up, there is no harm in doing so. But do not expect a sudden surge in AI-driven traffic or "better" citations in LLM responses simply by adding this file. The industry is still in the early stages of standardization, and the data shows the bots aren't looking for the map yet.