llms.txt: A Proposed Guide File for AI-Friendly Sites

For 20 years, robots.txt was the way to communicate with search engine crawlers. You'd place it at your site root to tell Google and Bing which pages to crawl and which to skip. Now, AI assistants from OpenAI, Anthropic, Google, and others read the web too, and they have different needs. llms.txt is a proposal for communicating with them.

The llms.txt proposal, published in 2024 by Jeremy Howard, co-founder of Answer.AI, is designed to help language models understand what your site is about and find its most useful pages. Rather than forcing an AI system to guess whether your site is an e-commerce store, a news publication, or a technical documentation site, llms.txt tells it explicitly. In principle, this clarity could help AI systems retrieve and cite your content more accurately. It remains a proposal, not an adopted standard, and no AI provider has committed to using it.

The Problem llms.txt Solves

AI systems face a practical problem. A language model can only read a limited amount of text at once, and turning a full website, with its menus, scripts, and ads, into clean usable text is hard. When ChatGPT or Gemini needs to answer a question about your site or product, it doesn't know what to prioritize. Should it read product pages or blog posts? Which documentation page is the reference? What pages are evergreen, and what pages are outdated? Without guidance, an AI system can waste its limited context on low-value pages or miss important content.

robots.txt helped with a different problem for search engines. It let you block URLs, point crawlers to your sitemap, and protect your crawl budget. But robots.txt only controls access; it doesn't explain your content. AI systems have different needs. They care less about ranking position and more about understanding what you're authoritative in.

llms.txt aims to bridge this gap. It gives language models a short summary of your site and a curated list of your key pages. Instead of an AI system having to infer that your site is about "SaaS billing solutions," you can tell it directly. The goal is to make your best content easier to find and use. There is no evidence yet that it improves indexing or the likelihood of being cited in AI answers.

Core llms.txt Structure and Syntax

The llms.txt format is plain Markdown, readable by both humans and language models. The file lives at your domain root (www.example.com/llms.txt) and follows a fixed order: a title, a short summary, optional details, and sections of links to your key pages. Here's a basic example:

# Example SaaS Company

> Billing automation software for B2B SaaS companies, covering pricing strategies, metering, payment processing, and compliance.

Example SaaS Company helps finance teams automate invoicing. The links below point to our most useful pages.

## Docs

- [Getting started](https://www.example.com/docs/getting-started): Set up your first billing plan
- [API reference](https://www.example.com/docs/api): Endpoints and authentication

## Guides

- [SaaS pricing strategies](https://www.example.com/blog/saas-pricing): How to choose and test a pricing model

## Optional

- [About us](https://www.example.com/about)

The syntax is intentionally simple so that both people and language models can read it. You state who you are, summarize what your site covers, and point to the pages that matter most. Unlike robots.txt, llms.txt contains no allow or block directives: it is a guide, not an access rule, and no major AI provider has committed to following it.

The Building Blocks of an llms.txt File

H1 title (the only required element) is the name of your site or project. Keep it concise and descriptive. "Example SaaS Company" is better than "Welcome to our site."

Blockquote summary is your elevator pitch for what the site covers. Be specific about your topical expertise. Instead of "We write about tech," write "We publish technical guides for Python developers, focusing on async programming, testing, and production deployment." This specificity helps AI systems understand your authority.

Optional details are paragraphs or lists, without headings, that add the context a model needs to read the rest of the file, such as who the site is for or how to interpret the linked pages.

H2 sections group your links by theme, such as Docs, Products, or Guides. Each section is a Markdown list of links to your key pages. Choose the pages you would want a model to read first.

Link items use the format [Page name](URL), optionally followed by a colon and a short description of the page. Use the canonical URL of each page, in the version you prefer (with or without www).

The Optional section is a special H2 titled "Optional". The links under it can be skipped when a shorter context is needed, so this is where secondary pages belong.

What is not in the file: llms.txt has no Allow, Disallow, Crawl-Delay, or date fields. It does not control access or crawling; those rules stay in robots.txt.

Going Further: Markdown Versions of Your Pages

Beyond the basic file, the proposal also suggests offering a clean Markdown version of each page that is useful to language models, available at the same URL with .md added (for example, www.example.com/docs/getting-started.md). The links in your llms.txt can then point directly to these versions:

- [Getting started](https://www.example.com/docs/getting-started.md): Set up your first billing plan

Some sites also publish an llms-full.txt file that gathers the full text of their key pages in a single file, for tools that want everything in one request. This is a convention used by some documentation sites, not a required part of the proposal. In both cases the goal is the same: less noise for a model to read.

llms.txt vs. robots.txt: Key Differences

robots.txt is primarily restrictive. You tell crawlers where they're NOT allowed. llms.txt is primarily informative. You tell language models what you're about and which pages matter. robots.txt uses a User-agent field to target specific crawlers, and major crawlers follow it; llms.txt has no such field, and major AI providers have not confirmed that they read it.

robots.txt affects search visibility directly. If you disallow crawlers from a page, it generally won't be crawled. llms.txt has no effect on traditional rankings, and its effect on AI visibility is unproven. If you use it, keep both files on your site, each with its own role.

In practice, use robots.txt to decide which crawlers, including AI crawlers such as GPTBot or ClaudeBot, can access which parts of your site. Use llms.txt only to point models toward your best content; it cannot restrict anything. For example:

robots.txt: blocks /checkout and /admin for the crawlers you choose (for example GPTBot)
llms.txt: links only to the docs, guides, and product pages you want models to read

Implementation Best Practices

Create your llms.txt file and place it at www.example.com/llms.txt. Use plain UTF-8 text with Markdown formatting. Make sure your web server serves it with a Content-Type header of text/plain or text/markdown. Test it by visiting the URL directly in your browser; you should see the raw text. Check that the structure follows the llms.txt specification: an H1 title, a blockquote summary, and H2 sections of links.

Write clear, specific descriptions. Don't just copy your homepage tagline. Be honest about what your site covers. If you publish content on 15 different topics, list them. If you're narrowly focused, say so. Honest, specific descriptions are more useful to a model than vague ones. Include keywords that describe your vertical or industry. If you're an e-commerce site, mention "e-commerce, products, pricing." If you're a SaaS company, mention "software, billing, integrations."

Update the file whenever you make changes to your key pages. The format has no date field, so keeping the file current is up to you. If your site's topical focus changes significantly, update the summary and the sections. Set a quarterly reminder to review and refresh your llms.txt, especially if you're creating new content categories or refining your positioning.

Monitor llms.txt adoption. Some documentation platforms and developer tools already publish llms.txt files, but OpenAI, Anthropic, and Google DeepMind have not committed to reading it when they crawl the web. Adding one is a low-cost experiment, not a guarantee of anything.

llms.txt and Privacy Concerns

Some people wonder whether llms.txt can control how AI companies use their content. It cannot. AI companies and privacy advocates are debating the ethics of web crawling and model training. Search and crawling standards have evolved over two decades to balance access with respect for content creators. llms.txt is part of this conversation, but it was designed to help models find your content, not to restrict how they use it.

If you want to prevent your content from being used for AI training, llms.txt is the wrong tool: it has no directive for that. Instead, use robots.txt rules for the relevant AI user agents, for example:

User-agent: GPTBot
Disallow: /

Compliance with robots.txt is voluntary too, but reputable AI providers publish the names of their crawlers and say that they respect it. If you want stronger protection, block bots in your server or firewall configuration. For now, neither file is a legal mechanism. As regulation develops and industry standards harden, more robust mechanisms may emerge.

Measuring llms.txt Impact

It's hard to measure ROI from llms.txt, since no AI provider has said that it uses the file. But you can track indicators. Monitor your AI mentions and citations across ChatGPT, Gemini, Claude, and Perplexity before and after you add the file. If you see citation growth, that is a correlation, not proof that the file caused it, because many other factors influence citations. Compare your citation growth to competitors who haven't implemented llms.txt.

Use AI mention tracking tools to quantify your AI search visibility. Track how many times your content is cited by major AI assistants. This lets you judge for yourself whether adding llms.txt changes anything for your site.

The Future of llms.txt

robots.txt was only formalized as an internet standard (RFC 9309) in 2022, after decades of use. llms.txt is still at the proposal stage, and it may or may not follow the same path. Sites without it are not penalized today, and no major AI search platform has said that it checks for the file.

Because creating the file takes minutes and costs nothing, adding an llms.txt is a reasonable low-risk experiment, especially for documentation-heavy sites. Just keep your expectations realistic: it is a proposal, its benefits are unproven, and your time is usually better spent on content quality, structure, and the signals AI systems actually rely on.

How to add an llms.txt file on WordPress, Shopify, Webflow and other CMS

The llms.txt file must answer at the root of your domain, at yourdomain.com/llms.txt, as plain text. How you get it there depends on how much control your CMS gives you over root files:

• Self-hosted sites and WordPress: upload the file to the root folder through your host's file manager or FTP. Some WordPress SEO plugins can also generate and serve it for you.
• Hosted platforms that block root uploads (many site builders and e-commerce platforms): use the platform's own llms.txt setting or app if it offers one; otherwise host the file elsewhere and add a redirect from /llms.txt, then check that the final URL still returns plain text.
• Static and headless sites: place llms.txt in the public folder so it ships with each deployment, and regenerate it when key pages change.

Write the content with the free llms.txt generator, then confirm the URL, content type and Markdown structure with the llms.txt validator. Our step-by-step installation guide details each case.

Conclusion

llms.txt is a proposed, low-cost guide file for sites that want to help language models find their best content. By placing this simple Markdown file at your domain root, you describe what your site covers and link to your key pages. The format is straightforward and implementation takes minutes, but no AI provider has committed to using it, so treat it as an experiment rather than a requirement. If you're serious about being cited by ChatGPT, Claude, Gemini, and Perplexity, focus first on clear, well-structured content and strong signals across the web. Use Sorank's keyword research and discovery tools to identify which topics to highlight in your llms.txt file.