llms.txt: What It Is and Whether You Need One

llms.txt is a proposed file you place at the root of your domain that tells AI systems, in plain Markdown, which parts of your site are worth reading. Think of it as a curated table of contents written for a machine rather than a visitor.

It arrived because the economics of search changed. Pew Research found clicks to traditional results drop to 8 percent when an AI summary is present, against 15 percent without, and with 900 million weekly ChatGPT users a lot of reading now happens through an assistant. Site owners reasonably want some say in what those assistants read.

Whether llms.txt gives you that say is the honest question, and the answer is more qualified than most articles about it admit.

The Challenges It Tries to Solve

• Context windows are finite. An assistant cannot read your entire site, so it reads whatever it lands on, which may be a thin page rather than your definitive one.
• Navigation confuses machines. Menus, cookie banners and promotional blocks surround your actual content and dilute what a model extracts.
• Sitemaps do not rank anything. An XML sitemap lists every URL with no signal about which pages carry your real expertise.
• Documentation is scattered. For technical products especially, the useful material is spread across many pages with no single entry point.

What the File Actually Looks Like

It is deliberately simple: Markdown, at yourdomain.com/llms.txt, with an H1 for your site name, a blockquote summary, and linked sections grouping the pages that matter.

A working example for a consultancy site:

Some implementations add llms-full.txt containing the actual content rather than links, so an assistant can ingest everything in one fetch. That is more useful and considerably more work to keep current.

The Part Most Articles Skip

llms.txt is a community proposal. It is not published by a standards body, not required by any AI vendor, and not confirmed as a ranking or retrieval input by the major assistants.

Google has stated publicly that it does not use it. The other large providers have not committed to reading it either. Some developer tooling and smaller retrieval systems do consume it.

So the honest position is this: llms.txt is cheap insurance with unproven return. That is a reasonable thing to buy. It is not a reasonable thing to prioritize over work that demonstrably moves citations.

Where It Genuinely Helps Today

The clearest value is for technical products with real documentation. Developers routinely paste a docs URL into an assistant and ask it to explain an API. A well built llms-full.txt makes that answer accurate instead of hallucinated, which is a direct support and adoption benefit regardless of search.

It also helps sites where the important content is genuinely hard to find: large catalogues, sprawling knowledge bases, anything where the good pages are buried three levels deep.

For a ten page brochure site, it changes nothing. The assistant was going to find your five real pages anyway.

How to Write One That Is Actually Useful

If you decide to add the file, the difference between a useful llms.txt and a decorative one comes down to curation. The point is selection, so listing every page defeats it.

Include the pages that represent your genuine expertise: the definitive explanation of what you do, the documentation that answers real implementation questions, the pricing or specification pages a buyer needs. Leave out press releases, category archives, thin landing pages and anything you would be embarrassed to have quoted.

Write the one line description after each link as though you were telling a colleague why to open it. "Pricing page" is useless. "Current pricing bands including minimum order and lead times" tells a machine exactly when this page is the right source.

Keep it short. Twenty well chosen links beat two hundred, because the entire value of the file is the editorial judgement it encodes. If you cannot decide what to leave out, the file is not doing its job.

Then treat it as a maintained artifact rather than a one off. A curation file that points at pages you retired eighteen months ago is worse than no file, because it actively misdirects anything that reads it. Put it on the same review cycle as your sitemap.

How to Decide

 

Do These First Instead

If your time is limited, spend it in this order. First, confirm you are not blocking AI crawlers, because everything else is theoretical while access is denied. Second, make sure your content exists in the HTML rather than being assembled entirely by client side scripts. Third, rewrite your highest impression pages so each section answers its own heading in the opening sentence.

Those three produce measurable movement. llms.txt is what you add afterwards, on a quiet afternoon, because it costs half an hour and might matter later.

One caution on how you evaluate this over the next year. Adoption of a proposal like this is genuinely uncertain, and both failure modes cost you: dismissing it entirely means missing a cheap advantage if support arrives, while treating it as essential means spending real hours on something no major assistant currently reads. Thirty minutes now and a note in your review calendar handles both risks at once.

The broader lesson holds beyond this file: emerging standards attract attention out of proportion to their current effect, while unglamorous fundamentals keep deciding outcomes. If you want a straight answer on which of the two your site actually needs, tell me what you are working with.

Frequently Asked Questions (FAQs)

Is llms.txt an official standard?

No. It is a community proposal, not a specification published by a standards body and not a requirement from any AI vendor. Treat it as an optional convention that some tools read and many ignore.

Does llms.txt replace robots.txt?

No, and the two do different jobs. robots.txt controls crawler access and is universally respected. llms.txt is a curation hint that points to your most useful content. Keep robots.txt correct regardless.

Where does the llms.txt file go?

At the root of your domain, so it resolves at yourdomain.com/llms.txt. It is plain Markdown: a heading with your site name, a short description, then linked sections pointing to the pages you consider authoritative.

Will llms.txt get me cited more often?

There is no reliable evidence that it does on its own. Citations follow crawlable, specific, well structured content. Add llms.txt as a low cost hedge after that work, not instead of it.

What should I do instead if I only have limited time?

Confirm AI crawlers are not blocked, make sure your key pages render their content in HTML, and rewrite your highest impression pages to answer their headings directly. Those three actions produce measurable change; llms.txt currently does not.