llms.txt shows up in nearly every AEO checklist right now, and it's easy to see why: it sounds like the AI-search equivalent of a sitemap, something no serious site should skip. The honest answer is more complicated. It won't get you cited by ChatGPT, Claude, or Google AI Overviews, but it's also not a waste of your time. Here's what the file actually is, what the AI companies themselves say about it, and where it does (and doesn't) belong in your AI search strategy.
What llms.txt actually is
llms.txt was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, in September 2024 (Answer.AI). The pitch was narrow and practical: language models have limited context windows, and turning a full HTML page into clean text a model can use is, in the spec's own words, difficult and imprecise. llms.txt tries to solve that by giving a model a short, structured map of a site instead of asking it to parse the rendered page.
The format itself is minimal. It's a single markdown file at your site's root (/llms.txt), built from one required H1 with the site or product name, an optional one-line blockquote summarizing what the site does, any additional context in a short paragraph, and one or more H2-headed sections listing links with a one-line description each. Some sites also publish a .md version of individual pages alongside the HTML, and an llms-full.txt that concatenates the whole site into one file for models that want everything at once. None of this is standardized by a search engine or AI lab. It's a community convention that's been widely written about, not officially adopted by anyone whose crawler decides what gets cited.
Does it actually help you get cited?
This is the part that gets skipped in most "how to set up llms.txt" guides: the evidence that anyone's reading it. Ahrefs analyzed llms.txt files across 137,000 sites and found that 97% were never fetched by an AI crawler at all (Ahrefs). Narrow that down to just the bots that actually determine AI citations, OAI-SearchBot for ChatGPT, PerplexityBot, and Claude's search crawler, and combined fetches across the entire sample numbered only in the hundreds.
That tracks with what the companies running those crawlers have said directly.
1. Google: explicitly says it's ignored
Google's Search Central AI optimization guide addresses llms.txt directly: these files "aren't needed to appear in Google Search," and keeping one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them" (Google Search Central). That covers both classic Search and AI Overviews. There's no ambiguity here: Google has told you, in writing, that it does not read the file.
2. OpenAI: no mention, no support
OpenAI has never documented llms.txt as something its crawlers use, and its publisher-facing documentation doesn't reference it. OAI-SearchBot, the crawler that determines ChatGPT search citations, indexes ordinary HTML the same way it always has. OpenAI does publish an llms.txt-format file for its own developer docs, but that's OpenAI using the format for its own documentation site, not a signal that ChatGPT's citation pipeline consumes other people's files.
3. Anthropic: recommends it, but for a different job
Anthropic is the most supportive of the three, and it's worth being precise about what that support actually covers. Anthropic's own engineering guidance recommends llms.txt-style documentation for AI agents and tools (Anthropic), and Anthropic publishes its developer docs in that format. But that's guidance for structuring documentation that agents, like Claude Code, reference while actively working with your API or SDK: a business-to-agent use case. It isn't confirmation that Claude's web search product reads llms.txt files when deciding what to cite in a conversation. Those are two different systems doing two different jobs, and it's easy to conflate them if you only read the headline.
So should you add one?
Given all that, the honest framing isn't "don't bother," it's "this won't move your AI citation numbers, but it also costs you almost nothing." Google's own wording is the clearest version of this: neither harm nor help. It's a single markdown file, maybe ten minutes of work, with no ongoing maintenance burden if you don't want one.
The one case where it's a genuinely good idea, not just a harmless one, is if you run a documentation or developer-facing site that AI coding agents actually consume. Claude Code, Cursor, and similar tools are built to look for exactly this kind of file when working with your API or SDK. That's a real, current use case, it's just a different one from getting AI-search citations, and conflating the two is where most of the "you need llms.txt for AI SEO" advice online goes wrong.
What to put in your llms.txt (if you make one)
If you decide it's worth the ten minutes, keep it simple and accurate rather than treating it like ad copy. There's no ranking algorithm reading it, so keyword density and persuasive language buy you nothing. If anything, an agent parsing the file cares more about it being a reliable map than a pitch.
A minimal, spec-accurate structure looks something like this:
# Acme
> Acme is a project management tool for small teams. This file helps AI agents find the right documentation for our product and API.
## Docs
- [Getting started](https://acme.example.com/docs/getting-started): How to set up your first project
- [API reference](https://acme.example.com/docs/api): Full REST API documentation
## Optional
- [Blog](https://acme.example.com/blog): Product updates and guides
The ## Optional section is part of the original spec's own convention: a place for lower-priority links a model can skip if it's working with a limited context budget. Stick to your most important pages, docs, pricing, core product pages, whatever best represents what your site actually does, and skip anything that reads like a sales pitch.
What actually moves the needle
If llms.txt isn't the lever, what is? The same fundamentals that show up across every AI platform we've looked at: content quality, with statistics and cited primary sources performing measurably better; clear structure, with direct answers and scannable headings; domain authority; and crawlability, making sure the bots that do determine citations, OAI-SearchBot, PerplexityBot, Google's crawlers, aren't accidentally blocked. We cover all four in detail, along with platform-specific checklists for Google AI Overviews, ChatGPT, and Claude, in our full guide to getting your website into AI search results.
Putting it together
llms.txt is a real, useful format for a narrower job than most of the advice around it suggests: helping AI agents navigate documentation, not helping your marketing site get cited in AI search answers. Add one if you run a docs site that AI coding tools will reference. Otherwise, it's optional homework: harmless if you do it, but not where the actual gains are. If you're trying to figure out why you're not showing up in a specific platform's answers, see our breakdowns for ChatGPT, Claude, and Google AI Overviews, or start with the full AI search optimization guide.
Tom Eastwood is the founder of Citation Hawk, a tool that monitors AI citation rates across ChatGPT, Claude, and Google AI Overviews.