llms.txt: What the File Is and Whether Your Site Needs One

This guide is based on the llms.txt proposal (version 2) and its list of changes, Google Search Central's page on AI features and your website, and the Chrome Lighthouse llms.txt audit, all checked on 8 November 2026. Where we give our own view, we say so.

What llms.txt is

llms.txt is a proposal by Jeremy Howard, first published in September 2024 and revised as version 2 in August 2026. It suggests adding a Markdown file at /llms.txt to "provide LLM-friendly content". The reasoning in the proposal is practical: HTML pages wrap their information in navigation, ads and JavaScript, which is hard to turn back into clean text, and an AI model's context window is too small to read most websites in full. A short file that summarises the site and points to the right pages saves the agent that work.

It is a proposal, not a web standard. The text says the specification "is open for community input" and lives in a public GitHub repository. Version 2 says thousands of sites now publish a file and that documentation platforms generate one automatically.

The file can sit at the site root or at any path. A file at /docs/llms.txt covers everything under /docs/, and where more than one file applies, agents should use the most specific one.

The file format

The file is Markdown, chosen because language models read it well, but it follows a fixed order so normal code can parse it too. The sections, in order:

Part	Required?	What it holds
H1 heading (# Name)	Yes, the only required part	The name of the project or site
Blockquote (> ...)	No	A short summary with the key information needed to understand the rest of the file
Paragraphs or lists	No	More detail about the site and how to read the linked files; any Markdown except headings
H2 sections (## Section)	No	"File lists": Markdown lists of links, each [name](url), optionally followed by : and a note
## Optional section	No, a convention	Secondary links an agent can skip when it needs a shorter context

An optional byte-order mark may come first. In version 1 the "Optional" heading told tools what to leave out; version 2 removed that special meaning, but kept the section as a useful convention.

The proposal's own advice for writing one: use concise, clear language, give each link a brief, informative description, avoid unexplained jargon, and test the file by asking an agent questions about your content with only the llms.txt as its starting point.

Markdown versions of pages

The proposal has a second part. Pages that agents might need should also be available as clean Markdown at the same URL with .md added (page.html.md) or with the extension replaced (page.md); for URLs that end in a folder, index.html.md or index.md. The links inside llms.txt should point to these LLM-friendly versions where they exist.

Version 2 adds discovery through standard link relations: rel="alternate" type="text/markdown" for a page's Markdown version and rel="describedby" for the llms.txt file that covers it, either as HTML <link> elements or an HTTP Link: header that can be set in the server or CDN without editing pages.

This part is the bigger job. A single llms.txt is a few minutes of writing; keeping a Markdown copy of every page in step with the HTML is a build or plugin task.

llms.txt, robots.txt and sitemaps

The three files are often mentioned together, but they do different things:

• robots.txt tells crawlers which URLs they may fetch. It is the file that controls access, including for AI crawlers; our guide to AI crawlers and your website covers the user agents and rules.
• sitemap.xml lists the indexable pages for search engines. See robots.txt and XML sitemaps for both.
• llms.txt is a curated overview read on demand. In the proposal's words, robots.txt lets automated tools know "what access to a site is considered acceptable", while llms.txt "is instead used on demand, when an agent needs information about a topic while assisting a user".

So llms.txt does not block or allow anything. Listing a page in it does not give any bot permission, and leaving a page out does not keep it private. If you want to limit AI crawlers, robots.txt (and the controls in the crawler guide) is still the tool. The proposal also explains why a sitemap is not a substitute: it often lacks LLM-readable versions, it does not include useful external links, and it usually covers far more than fits in a model's context.

What Google says

For Google Search, the answer is clear. Google's page on AI features says there are "no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary", and:

"You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add."

The same page says the foundational SEO practices apply: meet the technical requirements, follow Search policies and create helpful, people-first content. We found no Google Search documentation that says llms.txt is used for ranking or for choosing sources in AI Overviews. Our guide to AI Overviews and your website covers what does matter there and how the traffic shows in Search Console.

A separate Google product does check for the file. Chrome's Lighthouse has an llms.txt audit among its agentic browsing checks. It describes the file as "an emerging convention" and says that without it, agents "may spend more time crawling the site". The audit only flags a page when fetching llms.txt causes a server error; if the file is missing (a 404), the audit is marked Not Applicable, "as providing the file is optional at the moment". So a missing file does not fail Lighthouse, but a broken one does.

Who reads it

The proposal says llms.txt is used most heavily for software documentation, where coding agents follow it to find API references and tutorials, and that it has mainly been used at inference time (when an agent answers a user) rather than for training. It names AI labs that publish files for their own developer docs: OpenAI, Anthropic and Google's Gemini.

Publishing a file is not the same as reading other sites' files. We have not found a primary statement from a major AI search product saying its crawler or answer engine reads llms.txt on every site, so we do not claim it does. Agents can still fetch the file when a user or a tool points them at a site, which is the use the proposal describes.

Does a business site need one?

Our view, based on the sources above:

• It is not needed for Google Search or AI Overviews. Google says so directly. Spend that time on the page content, titles and technical health first.
• It is most useful for documentation. If you publish API docs, a help centre or product manuals that people use with coding or support agents, a file plus Markdown pages is worth doing.
• For a small business site it is cheap and low risk. A short file with your services, contact page, policies and FAQ takes little time. The proposal itself mentions "a business outlining its structure and policies" as a fit. Expect no measurable traffic change from it.
• If you publish one, keep it correct. An outdated file gives agents wrong prices, services or policies. Make sure it returns a 200 and not a server error, since that is what Lighthouse checks.

Several platforms generate the file automatically; the proposal lists Wix, Yoast SEO and AIOSEO for WordPress, Mintlify and GitBook. A generated file is a start, but check what it includes before relying on it.

An example for a small business

This is our own illustration of the format for a made-up firm, not a template from the proposal:

# Example Plumbing Co.

> Residential plumbing repair and installation in San Diego County. Open Monday to Saturday. Quotes by phone or email.

Prices on the site are starting prices; final quotes follow an inspection.

## Services

- [Drain cleaning](https://www.example.com/services/drains/): what is included and service area
- [Water heaters](https://www.example.com/services/water-heaters/): repair and replacement

## Company

- [Contact](https://www.example.com/contact/): phone, email, hours
- [Policies](https://www.example.com/policies/): warranty, cancellations, payment

## Optional

- [Blog](https://www.example.com/blog/): seasonal maintenance tips

Save it as plain UTF-8 text at /llms.txt, then open the URL in a browser to check it loads. Update it whenever you change services or policies.

Frequently asked questions

Does llms.txt help my site rank in Google?

Google's documentation says you don't need new machine-readable or AI text files to appear in AI Overviews or AI Mode, and we found no Google statement that llms.txt affects ranking.

Can llms.txt block AI crawlers?

No. It does not grant or deny access. Use robots.txt to set rules for crawlers; llms.txt only describes content for agents that choose to read it.

Where does the file go?

At the site root, /llms.txt, or at a path such as /docs/llms.txt, where it covers the pages under that path.

Is llms.txt an official standard?

No. It is a community proposal, now at version 2, maintained in a public GitHub repository and open for input.

Does Lighthouse fail my site without llms.txt?

No. Chrome's Lighthouse marks the audit Not Applicable when the file returns a 404. It flags the page only if requesting the file causes a server error.

Want your site to be easy for people, search engines and AI tools to read? See our website development and website maintenance services and all our services, or email [email protected].