CHABOT

`llms.txt`

5 minute read 1,163 words

Proposed by Jeremy Howard (Answer.AI, fast.ai) on September 3, 2024, llms.txt is a Markdown file at the root of a website, for example example.com/llms.txt. It provides an index and short descriptions of canonical content for tools that choose to read it.

Specification: llmstxt.org.

Origin and intent

Howard’s framing, in the original Answer.AI announcement, was:

  • Robots.txt and sitemaps were designed for search-engine crawlers.
  • LLMs need content structured for machine reading in addition to URLs.
  • A small, well-formed file at a canonical path can give LLMs a high-signal overview of what the site is about, what its key sections are, and where to find the canonical material.

The spec defines a Markdown file with:

  • An H1 title (the site / product name).
  • A blockquote summary.
  • Optional sections of links with short descriptions.

The specification is intentionally minimal, which makes it easy for documentation platforms to generate.

The “index + export” pattern

A pattern emerged quickly: two files, not one.

  • /llms.txt. A small index. Sub-10,000 tokens. Title, summary, lists of key pages with descriptions and links. Designed for real-time agents that need a fast overview.
  • /llms-full.txt. A fuller export. The complete documentation concatenated into a single Markdown file. Useful for ingestion pipelines, RAG systems, and IDE indexing.

Anthropic’s docs at docs.anthropic.com/llms.txt (8,364 tokens) and docs.anthropic.com/llms-full.txt (481,349 tokens) are the canonical example of this pattern.

Adoption

Documentation platforms adopted the convention quickly, while prominent sites adopted it more slowly and studies found mixed evidence of effect.

The Mintlify acceleration

In late 2024, Mintlify began auto-generating llms.txt and llms-full.txt for every docs site it hosted. Because its customers include Anthropic, Cursor, Resend, Perplexity, and other developer products, that change added the files to thousands of documentation sites.

Similar features rolled out at GitBook, Docusaurus plugins, ReadMe, Redocly, and Astro Starlight through 2025.

Adoption statistics

  • BuiltWith (October 2025) counted roughly 844,000 websites implementing llms.txt.
  • SE Ranking (2026) found ~10.13% adoption across crawled domains (≈ 300,000 domains in their dataset).
  • Rankability (June 2025) at one point measured only 0.3% adoption among top-1,000 sites, well below its estimate for the broader web. By 2026, that share had grown but high-authority domains still lagged smaller developer-focused sites.
  • ALLMO 2026 audit found that of the top 50 most-cited domains in major AI assistants, only one had llms.txt. Adoption is concentrated in mid-tier developer-product sites; large media and high-authority sites have not prioritised it.

Notable adopters (verified, mid-2026)

  • Anthropic. Claude docs publish both llms.txt and llms-full.txt.
  • Vercel. Across multiple product and documentation surfaces.
  • Stripe. Partial adoption on developer docs.
  • Cloudflare. Organised by product / service.
  • Mintlify. Authors the standard for its hosting customers.
  • Cursor. IDE-integration docs.
  • Hugging Face, Replicate, Perplexity. Versions on their docs sites.
  • NVIDIA. Split implementation (smaller llms.txt for tech docs; large file for main site).

Notably not yet (publicly) adopted by: OpenAI, Google, official Meta AI docs, Microsoft Learn (as of mid-2026).

The empirical question: does it work?

The available evidence is weaker than the marketing claims.

Vendors have claimed that llms.txt improves visibility in AI assistants and drives AI-mediated traffic. The evidence as of mid-2026 is:

  1. No major model vendor has publicly confirmed that their inference systems actively consume llms.txt as a primary signal. Anthropic, OpenAI, and Google have not issued statements that their assistants prefer or even reliably read llms.txt files.
  2. Citation studies have found mixed or null effects. Two notable 2026 studies (Signals.sh; SEO Engico) found no measurable lift in AI citations correlated with publishing llms.txt.
  3. Server-log evidence is thin. Some analyses of production server logs (Signals.sh) report essentially zero AI bots fetching llms.txt files specifically.

The current case for llms.txt is its low implementation cost and possible future utility, not a demonstrated increase in AI-mediated traffic.

Why DevRel teams still do it

Five reasons:

  1. Low cost. Auto-generated by Mintlify and similar; trivial to maintain.
  2. Documentation review. Producing the file requires an explicit index of canonical pages and can expose information-architecture problems.
  3. Possible future support. If assistants later adopt the convention, an existing file can be updated rather than introduced from scratch.
  4. Developer access. Developers and tool builders can use the file directly as a concise documentation index.
  5. Related maintenance. Teams that add llms.txt often also clean Markdown, page structure, and canonical URLs. Those changes should be evaluated separately from the file itself.

What a good llms.txt contains

Although the standard is minimal, the practical good version includes:

  • Clear product identity. What you are, in one sentence. Use the canonical name.
  • A short product description. Two to four sentences.
  • Sections organised by topic with short descriptions of each linked page.
  • Stable, canonical URLs. Don’t link to redirects; don’t link to deprecated paths.
  • Explicit dates and versions where applicable.
  • Cross-links to your llms-full.txt, OpenAPI spec, MCP server, and key sample-app repositories.

Examples worth studying (mid-2026):

  • docs.anthropic.com/llms.txt
  • nextjs.org/llms.txt
  • docs.stripe.com/llms.txt
  • developers.cloudflare.com/llms.txt

Common failure modes

  • Auto-generated and forgotten. Mintlify auto-generates one, but the team never reviews it. The file ends up listing dead pages or outdated names.
  • Marketing copy. Brand slogans and unsupported claims add no retrieval value and displace the factual descriptions the file is meant to index.
  • No llms-full.txt. The summary index is fine; the full export is missing. Ingestion pipelines that prefer full text can’t use the site.
  • Too large. Some llms-full.txt files become so large (hundreds of thousands of tokens) that retrieval systems silently truncate them.
  • Stale URLs. Site reorganisations break llms.txt links; the file goes uncorrected.

llms.txt is one of several agent-readable conventions. Others, varyingly mature:

  • OpenAPI / GraphQL. Standard for years; LLMs read these heavily.
  • MCP servers. Describe and expose product capabilities rather than documentation. See MCP for DevRel.
  • /.well-known/ URLs. Various conventions (security.txt, ai.txt) for site-level signals.
  • Schema.org / JSON-LD markup. Embedded structured metadata for pages and products.
  • Sitemaps.xml. Still useful; AI crawlers read them.
  • robots.txt. Often misconfigured; ensure GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended are explicitly allowed for the surfaces you want AI to read.

A product may publish clean information architecture, llms.txt and llms-full.txt, an OpenAPI specification, an MCP server, schema.org markup, an accurate sitemap, deliberate robots.txt rules, and transcripts for videos that contain unique technical material. Which elements apply depends on the product and on measured retrieval behaviour.

See also

Primary sources

  • Jeremy Howard (Answer.AI), “/llms.txt — a proposal to provide information to help LLMs use your website,” September 3, 2024.
  • The llms.txt specification at llmstxt.org.
  • Mintlify documentation and blog posts on llms.txt implementation.
  • Signals.sh, “Does llms.txt actually work?” (2026).
  • SE Ranking and Rankability adoption tracking, 2025 to 2026.