llms.txt Gets Zero Crawler Visits. We Ship It Anyway.

In April 2026, The SEO Framework published one of the more useful measurements of llms.txt to date: across six months of logged AI crawler traffic, covering 57 distinct AI bots and 180,000 AI-bot hits, the number of requests for llms.txt was zero.

Over a sample of that size, zero is a finding in its own right rather than a gap in the data.

Most WordPress SEO plugins now generate an llms.txt file and list it as an AI feature. WP Visibility generates it too. The difference lies in the help text beside the toggle, and that help text is the subject of this post.

What llms.txt is

llms.txt is a proposal published by Jeremy Howard of Answer.AI in September 2024 (llmstxt.org). The idea is a markdown file at /llms.txt that serves as a curated index of a site’s most important content, written for language models rather than browsers. It can be paired with .md versions of key pages so a language model can read those in place of the HTML.

It is a reasonable idea. As of June 2026 it remains a community proposal: no standards body has adopted it, and no major search engine or AI company has committed to fetching it.

The adoption story: one side of the market is growing

llms.txt adoption looks healthy until you ask which side of the market is growing.

The publishing side grew fast because SEO plugins made the file a checkbox. Yoast generates llms.txt. Rank Math generates llms.txt. Standalone plugins exist that do nothing else. Millions of sites can now produce the file with one click, so millions of sites do.

The consumption side is where the picture changes. A file format matters only if something reads it, and when the growth comes almost entirely from the publishing side, the file is being produced far more often than it is being read.

What the logs show

The consumption evidence, with names and dates, runs as follows.

The SEO Framework, April 2026. Six months of logs covering 57 AI bots and 180,000 AI-bot hits produced zero requests for llms.txt.

Ahrefs, June 2026. A log analysis of 137,000 domains found that 97 percent of llms.txt files got no requests at all in May 2026, and that AI retrieval bots accounted for 1.1 percent of the requests to the files that were fetched. Two independent datasets arrive at the same conclusion.

Google, on the record twice. Gary Illyes said at Search Central Live in July 2025 that Google does not use llms.txt and is not planning to. John Mueller has compared llms.txt to the old meta keywords tag, a piece of markup that publishers filled in and search engines ignored.

Google’s official documentation. The AI features guidance (last updated December 10, 2025) puts it in one sentence: “You don’t need to create new machine readable files, AI text files, or markup… There’s also no special schema.org structured data that you need to add.” Being indexed and eligible for snippets is the whole requirement for AI Overviews and AI Mode.

OpenAI’s crawler documentation. OpenAI’s bot docs describe one control for GPTBot and OAI-SearchBot: robots.txt, with published IP ranges. The only mention of llms.txt is a site-wide navigation link to OpenAI’s own documentation index, not a claim that its crawlers fetch yours.

Anthropic’s crawler documentation. The same picture: robots.txt, crawl-delay, published bot IPs, and no mention of llms.txt. The claim that “Claude reads llms.txt” still circulates, usually because Anthropic publishes an llms.txt for its own documentation site. Publishing a file for your own site says nothing about whether anyone fetches yours.

The best argument on the other side

The strongest case for llms.txt comes from Joost de Valk, Yoast’s founder, in an April 13, 2026 essay: “Standards don’t prove themselves. People prove them by building.” His argument inverts the usage data: a crawler has no reason to request a file that few sites serve yet, so publishers move first and demand follows. “Someone has to go first.” He points to XML sitemaps (2005) and IndexNow (2021), both of which began with almost no support and became standard infrastructure.

He has a point. There are real consumers today as well, though none of them are search engines. Perplexity has made supportive statements, and developer tools such as Cursor and Copilot, along with retrieval systems people build themselves, will read an llms.txt file when pointed at it.

The fair position is a modest one. llms.txt is no scam; it costs almost nothing to publish and, for now, returns almost nothing. Both halves of that sentence belong in the product.

Why the checkbox is everywhere

Many plugins generate the file because it is cheap to produce, and cheap to produce is a low bar to clear.

Every SEO plugin buyer in 2026 asks some version of “what are you doing about AI search.” It is a hard question for any plugin. A full answer takes real work: controls for each AI crawler, page information search engines can read, and a way to let an AI assistant work with your site. A checkbox that writes a markdown file is an afternoon of work, costs nothing to run, and fills a row in a feature comparison.

The file itself is not the problem; we ship it too. The trouble comes from framing it as the thing standing between a site and AI visibility when the consumption data shows near zero and Google says in writing that it reads no such file.

Why we ship it too

Three reasons, in order of weight.

1. It costs almost nothing. One toggle generates the file from content you already have. There is no performance cost, no maintenance burden, and no Search Console noise.
2. It might matter someday. Joost’s sitemap argument is right about one thing: if a major reader ever commits to llms.txt, the sites already publishing it are ahead by one toggle. Optionality at that price is defensible in a way that conviction would not be.
3. Parity. Buyers compare feature lists, and we would rather not lose a row over a file that takes nothing to produce.

The toggle in WP Visibility is labeled Experimental, and the help text under it reads, in full: “Experimental: no major AI crawler currently fetches llms.txt (verified June 2026).”

The label is deliberate. A plugin that is plain about a feature that does nothing yet is easier to trust on the features that do.

What moves AI visibility today

The part of the llms.txt story that tends to get lost is that a real set of AI-visibility controls already exists, documented by the companies themselves, and it involves no new file.

It begins with robots.txt, used with care. The 2026 bot population splits into three groups: training crawlers (GPTBot, ClaudeBot), AI-search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and fetchers triggered by a user. Block a training bot and you opt out of model training. Block an AI-search indexer and you disappear from ChatGPT, Claude, and Perplexity answers, which is what the blunt “block AI” toggles from 2023 quietly do to the sites that use them.

That, together with Google’s snippet controls and an IndexNow ping into the Bing index on which ChatGPT search is built, is the documented, supported, measurable set of controls. WP Visibility puts its work there, with per-bot rules and a warning before any rule that would remove you from AI answers, and puts one plain toggle on llms.txt.

The takeaway

llms.txt is worth having as a free option and worth being honest about. For now it is a file almost nothing reads, generated by almost every plugin, and rejected in writing by Google. It may be read widely one day. Until then, WP Visibility ships it, labels it Experimental, and tells you what the logs show.

If you want to see what AI crawlers can read on your site today, the free scan at wpvisibility.com grades your site’s AI visibility in about a minute. For more on features that plugins keep shipping after the platform moves on, see the companion post: 16 schema types Google deprecated that plugins still ship.

WP Visibility is $99 a year, with every feature, unlimited sites, client sites included, no account, and no tracking. See pricing.