Generate an llms.txt file for your website

What is an llms.txt file?

An llms.txt file is a Markdown document served at the root of a domain, at /llms.txt, describing what a website is about and which pages are worth reading. The convention was proposed at llmstxt.org as a way to give language models a short, curated map of a site instead of leaving them to infer one from navigation and boilerplate.

The reasoning is simple. A model answering a question about your product has a limited context window. Left to crawl, it spends that budget on menus, cookie notices, and footers. A curated file points it at your documentation, your pricing, and your key articles, in your own words.

Before using any llms.txt generator it is worth being straight about the status of this: llms.txt is a community proposal, not a standard any AI vendor has committed to. Publishing one is cheap and reversible, and several documentation platforms already generate them, but nobody guarantees it will be read.

How this llms.txt generator works

Enter a domain and the generator does three things.

1. Finds your pages

It looks for sitemap declarations in robots.txt first, then falls back to common locations such as /sitemap.xml and /sitemap_index.xml, following sitemap indexes within bounded limits. This is the same discovery the XML Sitemap Checker uses.

2. Checks each URL is worth listing

Every discovered URL is inspected before it reaches the file. Pages that return a 4xx or 5xx response, redirect somewhere else, or declare noindex are dropped and counted in the notes. This is the step most generators skip, and it is why their output frequently lists pages that no longer exist.

3. Reads the page for its real title and description

Each surviving page is fetched and its own title and meta description are used. Repeated brand suffixes are removed, because thirty consecutive lines ending in the same site name is noise when the name is already the heading of the file.

The llms.txt file format

The shape is deliberately plain Markdown, so a person and a model can both read it:

# Site name

> One sentence describing what the site is and who it is for.

## Pages

- [Page title](https://example.com/page): What the page covers.
- [Another page](https://example.com/other): What that one covers.

An H1 with the site name, a blockquote summary, then sections of links with a short note after each. Group links under further ## headings when a site has distinct areas such as documentation, guides, and product pages. Keep descriptions to a single clause; this is a map, not a brochure.

llms.txt vs robots.txt vs sitemap.xml

These three files are often confused because they all sit at the root of a domain and all concern crawlers. They answer different questions.

robots.txt is about permission. It states which user agents may request which paths, and it is the file that decides whether an AI crawler such as GPTBot or ClaudeBot may reach your pages at all. If robots.txt blocks a crawler, no llms.txt will persuade it otherwise.

sitemap.xml is about completeness. It lists every URL a site wants discovered, in machine-readable XML, with no notion of which pages matter more. It is written for search engine crawlers rather than for reading.

llms.txt is about emphasis. It is a short, human-readable, curated selection saying "this is what we do, and these are the pages that explain it". A sitemap with four thousand URLs and an llms.txt with twenty are both correct; they are answering different questions.

Publishing llms.txt does not replace either of the others. Keep your sitemap for discovery, keep robots.txt for access, and check what your robots rules actually allow before assuming a model can read anything at all.

How to add llms.txt to your website

Generate the draft above, then edit it. The llms.txt generator gives you an accurate starting point from your real pages; only you know which twenty pages deserve to be there and how to describe them for someone who has never seen your product.

Save the result as llms.txt and upload it so it is reachable at https://yourdomain.com/llms.txt, served as text/plain. On a static site, that means the public or root directory. On WordPress, place it in the web root rather than inside a theme, since themes are replaced on update. On Shopify and other hosted platforms, use whichever file or redirect mechanism the platform exposes for root-level files.

Then confirm it loads by visiting the URL directly. A file that returns HTML, a 404, or a redirect to your homepage is not being served correctly, and this is the most common mistake after publishing.

Regenerate it when your site changes meaningfully. A file listing pages you deleted six months ago is worse than no file, because it actively points models at dead URLs.

llms.txt questions