What Is llms.txt? The Standard, Explained
llms.txt is a proposed markdown file at your site root that gives AI crawlers a curated map of your best content. The spec, the skeptics, and how to ship one fast.
On this page
llms.txt is a proposed standard: a plain markdown file served at your domain root — yoursite.com/llms.txt — that gives large language models a curated, annotated map of your most important content. Proposed by Answer.AI co-founder Jeremy Howard in September 2024, it is a reading list for AI systems rather than an access-control file: a short summary of who you are, followed by grouped links with one-line descriptions, ordered by what you most want a machine to read first.
Where did llms.txt come from?
The proposal came out of a practical annoyance. Jeremy Howard, co-founder of Answer.AI and creator of fast.ai, published the spec at llmstxt.org in September 2024 after watching language models struggle with a problem human readers never notice: websites are built for browsers. Navigation chrome, cookie banners, JavaScript-rendered content, and pagination all get in the way of a model trying to extract the substance of a site, and context windows are too small to ingest a whole domain anyway.
His answer borrowed the shape of robots.txt — a well-known filename at a predictable location — and filled it with the opposite intent. Where robots.txt says where crawlers may go, llms.txt says where they should go, in a format models natively parse well: markdown. The proposal also sketched two companions that matter more for documentation-heavy sites: markdown twins of individual pages (append .md to a URL and get the clean-text version) and an llms-full.txt that inlines complete content for models that want everything in one fetch.
The idea landed hardest in developer tooling, where the consumer was obvious: AI coding assistants that need current, accurate API documentation at inference time. From there it spread into marketing, where the pitch is the same one behind generative engine optimization: if AI assistants are becoming a discovery surface, being easy for them to read is worth something.
What does an llms.txt file contain?
The spec is deliberately small. A conforming file is ordinary markdown with a fixed skeleton:
# EGGKNITE
> Growth marketing and applied-AI agency. We run paid media,
> lifecycle, analytics, and AI search optimization for brands
> that judge marketing on contribution margin.
## Services
- [Paid Media](https://eggknite.com/services/paid-media): full-funnel
paid search and social with margin-first math
- [AI Search Optimization](https://eggknite.com/services/ai-search-optimization):
visibility in ChatGPT, Perplexity, and AI Overviews
## Resources
- [ROAS Calculator](https://eggknite.com/calculators/roas): break-even
ROAS from your real contribution margin
## Optional
- [Blog](https://eggknite.com/blog): the full article library
Four elements do all the work. The H1 names the site or project — the only strictly required piece. The blockquote gives a one-paragraph summary a model can use to orient itself. The H2 sections group links, each with a one-line description of what the target page actually contains; that annotation is the part a bare sitemap can never offer. The Optional section has a defined meaning in the spec: content a model may skip when its context budget is tight, which makes everything above it an explicit priority claim.
Notice what the format asks of you editorially. Writing a good llms.txt forces you to decide which ten to thirty pages define your business — a genuinely useful exercise even if a crawler never reads the result. Our step-by-step guide to writing llms.txt walks through the choices section by section.
How is llms.txt different from robots.txt and sitemap.xml?
The three root files get conflated constantly, and the confusion causes real mistakes — like teams believing llms.txt can block AI training crawlers. It cannot. Blocking is robots.txt's job.
| File | Job | Format | Consumed by |
|---|---|---|---|
| robots.txt | access control — what crawlers may fetch | directive syntax | search + AI crawlers (well established) |
| sitemap.xml | inventory — every indexable URL, no context | XML | search engines (well established) |
| llms.txt | curation — what matters most and why | markdown | LLM tools (proposed, adoption unproven) |
A useful mental model: robots.txt is the bouncer, sitemap.xml is the phone book, and llms.txt is the concierge's shortlist. They coexist without overlap, and llms.txt changes nothing about how the other two behave. It also does nothing that schema markup does — structured data annotates meaning inside pages for machines, while llms.txt points at whole pages from outside. Sites serious about machine readability generally want both layers.
Who uses llms.txt, and what do the skeptics say?
Publisher-side adoption is easy to verify: fetch the path. Anthropic serves one for its documentation, Cloudflare publishes one for its developer docs, and Mintlify's decision to auto-generate the file for every docs site it hosts pushed adoption across thousands of developer-tool companies in a single stroke. Community directories track hundreds more, from AI startups to marketing sites. Publishing the file has become table stakes in developer tooling and is spreading outward from there.
The consumer side is where the skeptics live, and their arguments deserve a fair hearing. No major AI provider — OpenAI, Anthropic, Google, Perplexity — has formally committed to fetching llms.txt as part of crawling or retrieval. Google has been the bluntest: its representatives have said Google's systems do not use the file, with John Mueller publicly comparing it to the old keywords meta tag — a signal site owners control entirely, which is exactly why engines learned to ignore such signals. Practitioners who watch server logs report that major AI crawlers request the file rarely, though AI coding tools and smaller answer engines fetch it opportunistically when a user points them at a domain.
Both sides can be right at once. The file is genuinely cheap to produce and genuinely unproven as a ranking or citation input. What tips the decision for most teams is asymmetry: the cost is an hour, the failure mode is an ignored text file, and the upside — being the easily-parsed source when an assistant needs one — compounds if any major consumer switches it on. You can watch whether that bet pays off by tracking your AI share of voice over time, and our free AI Visibility Checker gives you a fast read on how assistants currently see your brand.
Does llms.txt matter yet?
Here is the honest scorecard as of 2026. As a direct visibility lever, unproven: no controlled study shows llms.txt causing citations or mentions, and the biggest engines say they ignore it. As a hygiene item, cheap and sensible: the same category as a clean sitemap — something you ship once, keep current, and stop thinking about. As an editorial forcing function, quietly valuable: deciding which pages belong in the file is a prioritization exercise most marketing teams have never done explicitly.
What llms.txt is definitively worth less than: the fundamentals. Answer engines select sources on structure, clarity, freshness, and authority — the well-documented citability factors — and a manifest file cannot compensate for pages that are hard to parse or thin on substance. Technical hygiene layers like Core Web Vitals and structured data have measurable consumers today; llms.txt has hopeful ones. Sequence your effort accordingly: fundamentals first, manifest second. Both fit inside a broader AI-search program of the kind our AI search optimization practice runs end to end, from citability audits through measurement.
The realistic upside case runs through the long tail. Smaller answer engines, RAG pipelines, and agent frameworks adopt conventions faster than giants do, and several already look for llms.txt when they encounter a domain. If assistants keep gaining share of discovery — the trajectory every published dataset points to — conventions that make sites machine-legible acquire consumers over time rather than losing them.
How do you ship an llms.txt in under an hour?
The whole job is five steps, and none of them is technical enough to need an engineer.
- Shortlist the pages that define you. Ten to thirty URLs: core service or product pages, your best evergreen explainers, pricing, docs. Skip anything you would be embarrassed to have an AI quote.
- Write the H1 and blockquote. Name, then two or three sentences a stranger — or a model — could use to classify your business accurately.
- Group and annotate. Two to five H2 sections with links, each carrying a one-line description of what the page actually delivers. The descriptions are the product; write them like good meta descriptions.
- Add the Optional section. Blog indexes, archives, secondary material — present but explicitly skippable.
- Serve and maintain. Publish at /llms.txt as plain text, then recheck it quarterly or whenever key pages move. A stale manifest is worse than none, because it points machines at the wrong pages with your endorsement attached.
Our free llms.txt Generator turns a pasted list of URLs into a spec-conformant file in a few minutes, and the growth marketing glossary collects this term alongside every other definition in this series — from attribution to sender reputation — in one place.