What Is llms.txt? What It Does, Who Reads It, and Whether You Need One

llms.txt is a plain markdown file at the root of a website that gives AI agents a short summary of the site and a list of links to its most useful content. Jeremy Howard proposed it in September 2024, and a revised version, v2, followed in August 2026.

Whether you need one depends on who you expect to read it. It will not help you rank in Google, which says Google Search ignores the file. No AI assistant has said it reads llms.txt when deciding what to cite. It is genuinely useful to coding agents working from documentation, and it is a cheap way to state, in your own words, what your company is. This guide covers what the file is, what changed in v2, who actually reads it, and how to write one.

What is llms.txt?

The proposal starts from a practical problem. Web pages are built for people: the information is wrapped in navigation, ads and JavaScript, and converting it back into clean text is imprecise. Agents have limited context, and every wasted token costs time and money. llms.txt gives them one small file that says what the site is and where the useful material lives.

The format is ordinary markdown with a fixed order, so it can be read by a model or parsed by a simple program:

• An H1 with the name of the site or project. This is the only required part.
• A blockquote with a short summary of what the site is.
• Optional paragraphs or lists with anything else an agent should know before reading further.
• H2 sections of links, each link followed by an optional note on what it contains.

The file lives at /llms.txt. The proposal also asks sites to publish clean markdown versions of important pages, so that the links in llms.txt point to content an agent can read without stripping HTML.

What changed in llms.txt v2

The August 2026 revision is based on, in Howard's words, "two years of adoption." Most guides ranking for llms.txt were written before it and still describe v1. The changes that matter:

• How agents use the file is now stated. "Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links." The file is a map, not a content dump; it should stay small enough to fit in context.
• Discovery through link relations. A page can point to its markdown version with rel="alternate" type="text/markdown" and to the llms.txt that covers it with rel="describedby", either as HTML <link> elements or an HTTP Link header.
• Two markdown URL forms. v1 specified page.html.md. v2 also allows replacing the extension, page.md.
• Subpath files are defined. A file at /docs/llms.txt covers the pages under /docs/, and the most specific file applies.
• The "Optional" section lost its special meaning. v1 tooling used it to decide what to leave out when expanding the file into context. That tooling is no longer part of the proposal. Optional sections are still allowed as a convention for secondary links.

Howard also writes that thousands of sites now publish the file, that documentation platforms generate it automatically, and that OpenAI, Anthropic and Google publish llms.txt files for their own developer documentation.

Does Google use llms.txt?

No. Google's guide to optimizing for its generative AI features lists llms.txt among the things you can ignore: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."

The same page adds that creating one is fine for other systems that use it, and that doing so "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." That covers AI Overviews and AI Mode as well as ordinary results.

Chrome is a different story. Lighthouse includes an llms.txt audit in its agentic browsing checks, on the grounds that "without this file, agents may spend more time crawling the site to understand its high-level structure and primary content." The audit flags a server error when fetching the file; a missing file is marked not applicable. So Google's browser tooling treats llms.txt as agent readiness, while Google Search ignores it.

Do ChatGPT, Claude and other AI assistants read llms.txt?

No AI company has said that its search crawler, or its assistant when answering a question, uses llms.txt to decide what to read or cite. The labs do publish the files for their own developer documentation. OpenAI's crawler documentation is a good example: the page points readers to its own llms.txt and to markdown versions of its docs, while describing crawler controls only in terms of robots.txt.

That fits what the proposal itself says about where the file is used: "llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials."

It helps to separate two kinds of AI traffic:

Both kinds already read the pages you have. Our server logs show AI systems reading our ordinary HTML pages directly: ChatGPT fetching pages on a user's behalf, OpenAI's search crawler, Anthropic's Claude, Perplexity and Meta's AI crawler among them. None of them needs an llms.txt to reach your content. Crawlable, clearly written HTML still does that work.

Independent measurement points the same way. Ahrefs reported that of roughly 38,000 domains in its study with a valid llms.txt, 97% received zero requests for it in May 2026.

Should you create an llms.txt file?

We added an llms.txt to thrad.ai in September 2026 for one reason: to put our own description of Thrad in front of AI systems. Assistants tended to describe us as only one side of the business, an ad network for advertisers or a monetization SDK for publishers, when we run both. The file opens with a summary that says so. We have not measured whether assistants' descriptions have changed since, and we would not claim they have.

That is the job we would suggest for a marketing site: a correct, current description of what you are. Treat it as maintenance, not a visibility tactic.

How to write an llms.txt file

1. Open with an H1 and a one-paragraph blockquote saying what you do and for whom. It is the part an agent is most likely to read, so put the thing you most want it to get right here.
2. Add a short paragraph for facts agents get wrong: what you are not, who you serve, how you charge.
3. Group links under H2 headings, each with a one-line note, and keep the file small. Detail belongs behind the links.
4. Point links at clean content. Markdown versions if you publish them; otherwise your plainest HTML pages.
5. Serve it at /llms.txt as plain text with a 200 response. Lighthouse flags server errors.
6. Generate it from the same data as your site navigation, so it cannot drift out of date when pages move.
7. If you publish markdown versions of pages, advertise them with rel="alternate" type="text/markdown" and point to the file with rel="describedby", as v2 describes.

A minimal file for a fictional company:

# Acme Analytics

> Product analytics for mobile apps: events, funnels and A/B tests.

Acme is not an ad or attribution tool. Pricing is per tracked user.

## Product

- [Features](https://acme.example/features.md): What the product does
- [Pricing](https://acme.example/pricing.md): Plans and limits

## Docs

- [Quickstart](https://acme.example/docs/quickstart.md): Send a first event
- [API](https://acme.example/docs/api.md): Endpoints and rate limits

## Company

- [Security](https://acme.example/security.md): Data handling

Documentation platforms such as Mintlify generate the file automatically, along with an llms-full.txt that contains the full documentation in one file. llms-full.txt is a platform convention; the proposal does not define it.

How to see who reads your llms.txt

The answer lives in your server or CDN logs. Filter them to requests for /llms.txt and group by user agent.

Check what your log pipeline keeps. For our file's first month we could not see who requested it at all: our log pipeline kept page requests and dropped text files. We now log those too. Most sites are in the same position, since browser analytics never sees a request for a text file.

When you do have the data, compare requests for llms.txt with requests for your HTML pages from the same agents. That tells you more about how AI systems use your site than the llms.txt count alone.

The bottom line

llms.txt is a small, cheap file with a narrow job. For documentation it is worth doing. For a marketing site, it is a place to state plainly what you are, not a way into search results or AI answers. Getting cited still depends on crawlable pages worth citing; our guides to ranking in ChatGPT and GEO versus SEO cover that work.

Some brands would rather not wait to be cited. Thrad Autopilot places clearly labeled ads inside AI conversations, matched to what people are asking.