llms.txt
llms.txt is a proposed standard for a Markdown file, usually served at /llms.txt, that gives large language models a short summary of a website and a curated list of links to the pages that matter most.
What is llms.txt?
llms.txt is a site guide written for language models rather than for search crawlers. The llms.txt proposal was published by Jeremy Howard on 3 September 2024.
The idea is simple. Web pages carry navigation, scripts and ads, and a model's context window is limited. A small Markdown file can point an AI tool straight to clean, useful content.
It is a proposal, not a formal web standard, and it does not control crawling.
What goes in an llms.txt file?
Only one element is required: an H1 with the name of the project or site. The proposal lists the parts in this order:
An H1 with the site or project name (required).
A blockquote with a short summary (optional).
Sections of plain text with more detail, without headings (optional).
H2 sections with lists of links, each written as [name](url) with an optional note after a colon (optional).
An H2 named "Optional" for secondary links a tool can skip when it needs less context.
The file can sit at the site root or under a path such as /docs/llms.txt. The proposal also suggests a clean Markdown copy of key pages at the same URL with .md added.
What does an llms.txt file look like in practice?
A good llms.txt is short, curated and kept in sync with the site. Picture a B2B software company with 400 help articles.
The docs lead writes an H1 with the product name and a one-line blockquote saying what it does and for whom.
She adds an H2 called Docs with the 15 articles that answer most support tickets, each with a one-line note.
An H2 called Pricing and policies links the pricing page, the security overview and the terms.
An H2 called Optional lists the changelog and older API versions.
The build script regenerates the file when docs change, so links do not break.
Say a developer gives the llms.txt URL to a coding assistant. The assistant now reads 15 clean pages instead of guessing from the home page.
What are the limits of llms.txt?
Google says you do not need AI text files to appear in its AI features, so llms.txt is not a ranking lever.
Google's position: its guidance on AI features says sites "don't need to create new machine readable files, AI text files, or markup" to appear in AI Overviews or AI Mode.
Uneven use: which AI tools read the file varies. Check your server logs for real requests before you spend much time on it.
Drift: a hand-written file goes stale as pages move.
Not access control: it does not allow or block crawlers.
Its clearest use today is for AI assistants and developer tools that a user points at your docs or site.
How is llms.txt different from robots.txt and sitemap.xml?
robots.txt sets crawl rules, sitemap.xml lists every URL for search engines, and llms.txt picks the few pages a language model should read first. robots.txt follows the Robots Exclusion Protocol (RFC 9309, 2022), which itself says its rules "are not a form of access authorization."
A sitemap is an XML file that lists a site's URLs so search engines can find and crawl them. llms.txt is about curation and context, not completeness.
The three work together: keep robots.txt and your sitemap for search, and add llms.txt if people use AI tools on your content. It sits next to answer engine optimization and GEO, and is close in spirit to the curated sources used in retrieval-augmented generation.
llms.txt vs robots.txt
llms.txt | robots.txt | |
|---|---|---|
Purpose | Point language models to key content | Tell crawlers which paths they may fetch |
Format | Markdown | Plain-text rules |
Status | Proposal published in 2024 | IETF standard, RFC 9309 (2022) |
Effect on crawling | None | Allows or disallows paths for compliant crawlers |
Audience | AI assistants and agents | Search and other crawlers |
How does Vizio AI approach llms.txt?
Vizio AI publishes its own llms.txt but has no published client case study on it. The file at www.vizio.ai/llms.txt follows the proposal: an H1, a one-line summary, then H2 sections that link the studios, solutions and other key pages, with a pointer to the sitemap for the full URL list.
Where to start: write a short llms.txt with your product name, a one-sentence summary and links to the ten pages that answer most customer questions, then review it each quarter.