AI search glossary
What Is llms.txt, and Do AI Tools Actually Read It?
Plenty of audits now say “add llms.txt for AI SEO”. This entry covers the v2 format, what changed in August 2026, who reads the file, and Google’s plain statement that Search ignores it.
What is llms.txt? The short answer
llms.txt is a proposed Markdown file, usually at /llms.txt, that gives AI agents a short, curated guide to a website: a title, a one-line summary and lists of links to the pages that matter. Jeremy Howard proposed it in September 2024. It is optional, and Google says Search ignores it.
Key takeaways
- The only required part is an H1 with the site’s name. Everything else is optional.
- Version 2 (August 2026) lets a file cover one path and adds standard link relations.
- Google’s July 2026 guide says Search ignores the file. It neither helps nor harms rankings.
- Agents use it on demand, mostly for software documentation, according to the spec.
- List only public, canonical pages. The file is readable by anyone.
# Digital Magma
> Digital Magma is a digital marketing agency based in Noida,
> Uttar Pradesh, India. Its public website covers SEO, local SEO,
> AI search visibility, Google Ads and more.
- Founder: Ankit Sachan.
- Homepage: https://digitalmagma.com/
## AI search visibility
- [ChatGPT SEO](https://digitalmagma.com/seo-services/chatgpt-seo/): OAI-SearchBot access, entity facts, answer-first pages
## Optional
- [About us](https://digitalmagma.com/about-us/)llms.txt: how it works and what the format looks like
llms.txt works like a table of contents written for language models. An agent reads it first, then follows only the links it needs.
The spec at llmstxt.org explains the reason. HTML pages wrap facts in menus, ads and scripts, and context windows are still too small for most whole sites.
A file that follows the spec uses Markdown sections in a fixed order:
- An H1 with the name of the site or project. This is the only required section.
- A blockquote with a short summary and the key facts needed to read the rest.
- Optional detail, such as paragraphs or lists, with no headings.
- H2 sections of “file lists”. Each item is a Markdown link, optionally followed by a colon and notes.
A section named “Optional” holds secondary links an agent can skip when it needs a shorter context.
The spec also suggests a clean Markdown copy of important pages at the same URL. That means page.html.md, or page.md with the extension replaced.
The format is strict enough for parsers and regular expressions, yet plain enough for a person to read.
What changed in llms.txt v2 (August 2026)
llms.txt v2 came out in August 2026, after two years of adoption. The spec’s Changes page lists what moved.
| Topic | v1 (September 2024) | v2 (August 2026) |
|---|---|---|
| Location | Root, with subpaths allowed but undefined | A file covers every page under its path. The most specific file applies. |
| Markdown pages | page.html.md only | page.html.md or page.md |
| Discovery | Not specified | rel=”alternate” type=”text/markdown” and rel=”describedby”, as HTML link elements or an HTTP Link header |
| How agents use it | Expand the file into context with a tool | View or search the file, then follow the relevant links |
| “Optional” section | Told tools what to drop | Still allowed, as a convention only |
The path rule matters for shared hosts. A GitHub Pages project, for example, can publish /project/llms.txt even though it cannot touch the host’s root.
That is also why v2 did not move the file to /.well-known/. The spec notes that well-known URIs exist only at the origin root.
A header example from the spec:
Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"llms.txt vs robots.txt vs sitemap.xml vs llms-full.txt
llms.txt is often mistaken for a crawl rule. It is a guide, and it sits beside three other files with different jobs.
| File | Job | Defined by | Read by |
|---|---|---|---|
| robots.txt | Says which paths crawlers may fetch | IETF RFC 9309 | Search and AI crawlers that follow it |
| sitemap.xml | Lists every indexable URL | The sitemaps.org protocol | Search engines |
| llms.txt | Curated guide and key links for agents | llmstxt.org proposal | Agents and tools that choose to |
| llms-full.txt | Full content in one Markdown file | No spec. The llmstxt.org page does not define it. | Tools that choose to |
The spec says llms.txt “can complement robots.txt” by giving context for allowed content. It does not grant or block access.
A sitemap lists everything, often far too much for a model’s context, and rarely includes Markdown versions or useful outside links.
llms-full.txt is a convention, not part of the spec. OpenAI, Anthropic and Perplexity each publish one for their developer docs, beside their llms.txt.
llms.txt for a business: who reads it and who ignores it
llms.txt is read by some agents and ignored by Google Search. Both facts come from primary sources.
Google’s generative AI guide (updated 10 July 2026) lists llms.txt among things to ignore. It says the file will “neither harm nor help” visibility, “as Google Search ignores them”.
The same guide says it is “completely fine” to keep one for other services. So this is a choice about agents, not about Google rankings.
On the reading side, the spec says coding agents use llms.txt most, to find API references. The AI labs publish their own for developer docs: OpenAI, Anthropic and Gemini.
The spec also says the file mainly serves inference rather than training. An agent fetches it while helping a user.
For a service business, the honest value is small but real. That is how Digital Magma explains it to clients who ask. A short, accurate file hands any agent your name, services, prices and contact facts in one fetch.
llms.txt: what to do (create, link, validate and maintain)
Setting up llms.txt takes an afternoon. Digital Magma follows these four stages:
- Create. Write an H1, a one-paragraph blockquote and H2 lists of your most useful public pages. Add a short note to each link.
- Link. Serve it at /llms.txt as plain text. Optionally add a rel=”describedby” link or Link header, as v2 suggests.
- Validate. Open the file in a browser, check each link returns 200, then run Lighthouse’s agentic browsing audit.
- Maintain. Regenerate it whenever pages, prices or URLs change. A stale file misleads agents.
The spec adds one useful test. Give an agent only your llms.txt, then ask it questions about your business.
Chrome’s Lighthouse llms.txt audit flags a server error when fetching the file. A missing file (404) is marked Not Applicable, because the file is optional.
Our own digitalmagma.com/llms.txt has eight H2 sections, from services to an Optional list. Our robots.txt points to it in a comment line, not as a directive.
Digital Magma has not yet added the v2 describedby header or Markdown page copies. They are next on our list.
llms.txt mistakes: private links, stale URLs and ranking myths
Most llms.txt mistakes come from treating it as a secret or as an SEO lever. It is neither.
- Listing private URLs. Staging sites, admin paths and client folders do not belong in it. Anyone can read the file.
- Using it as a block list. It cannot stop a crawler. Access rules belong in robots.txt, and real protection needs a login.
- Letting links go stale. Redirected or deleted URLs waste an agent’s fetches and can repeat outdated prices.
- Expecting rankings. Google says Search ignores the file, so it will not move your positions.
- Dumping the sitemap. Hundreds of links defeat the point. Keep it curated and short enough for a context window.
RFC 9309 makes the same point about robots.txt: listing paths makes them public and easier to discover.
llms.txt: frequently asked questions
Short answers to the llms.txt questions site owners ask us.
What is llms.txt used for?
It gives AI agents a short guide to a site: what it is, and which pages hold the key facts. Agents read it while helping a user, then follow the links they need.
Does Google use llms.txt?
Google Search does not. Its July 2026 guide says Search ignores the file, so it neither helps nor harms rankings. Google says keeping one for other services is fine.
Is llms.txt the same as robots.txt?
They do different jobs. robots.txt tells crawlers which paths they may fetch. llms.txt offers context and links for agents, and it cannot allow or block anything.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short index of links. llms-full.txt, where it exists, puts full content in one Markdown file. Only llms.txt is defined at llmstxt.org.
Where should the llms.txt file go?
Put it at the site root, such as example.com/llms.txt. Under v2, a file at a subpath like /docs/llms.txt covers only the pages under /docs/.
Is there a safe llms.txt generator or validator?
The spec lists generators, including Yoast SEO, AIOSEO, Wix, Mintlify and GitBook. Whatever you use, read the output before publishing it, and check that no private URLs slipped in.
Does llms.txt improve SEO?
Not for Google. Google says Search ignores it. Its value lies with agents and tools that choose to read it, which is a separate goal from rankings.
Does Lighthouse check for llms.txt?
Lighthouse has an llms.txt audit among its agentic browsing checks. A server error fails it. A missing file is marked Not Applicable, so you are not penalised for skipping it.
llms.txt done right, in one afternoon
Digital Magma writes, validates and maintains llms.txt files, alongside robots.txt and AI crawler checks. See our LLMO agency service or book a free SEO audit.
No AI mention is guaranteed. Call +91 79823 19697, email contact@digitalmagma.com or message on WhatsApp (Mon–Sat, 9 am–9 pm IST). You can read our Google reviews before you get in touch.