Quick Summary
Seen the hype? “llms.txt is the new robots.txt!” Reality check: it’s a tiny Markdown file at yourdomain.com/llms.txt mapping your key pages, ten minutes to make. But no major AI provider confirms reading it, and Ahrefs found 97% of published files got zero requests in 2026. Cheap insurance but real effort belongs on crawler access and content.
Every few months, SEO Twitter finds a new file you’re REQUIRED to have or your site vanishes into the void. In 2026, it’s llms.txt. The claims are wild: “AI can’t find your site without it.” “ChatGPT reads this before your homepage.” “It’s the new robots.txt.”
Most of that? Hype. Pure hype.
But underneath there’s a reasonable idea and a genuinely interesting question about what AI crawlers do on your site. So here’s both sides, straight: the practical bit (what it is, how to make one in ten minutes) and the evidence bit (does the data say it matters). No guru nonsense. Just facts.
Key Takeaways
- The file itself is dead simple. Plain Markdown at /llms.txt. An H1 with your site name. A one-line summary. Then your key links. That’s it. Jeremy Howard (the Answer.AI guy) proposed it back in September 2024. v2 landed August 2026. Not complicated.
- Stop confusing it with robots.txt. They do OPPOSITE jobs. robots.txt is the bouncer it tells crawlers where they CAN’T go. llms.txt is like leaving a helpful map at the front desk “hey, these pages are worth reading.” You need robots.txt no matter what. llms.txt is optional.
- Here’s what the GEO gurus won’t tell you: as of late 2026, not ONE major AI provider has confirmed their crawlers fetch this file. Not one. Ahrefs checked 137,000 domains. 97% of published llms.txt files got literally zero requests. Let that sink in.
- Still worth ten minutes? Sure. But do it LAST, not first. Get your crawler access sorted. Publish good content. Add schema. Fix your speed. THEN drop in the file as the cherry on top. Not the cake. The cherry.
- The real recipe for AI visibility? Let the bots actually crawl you. Publish stuff worth quoting. Get mentioned in places AI already reads — Reddit, YouTube, the press. That’s it. No magic file required.
What llms.txt Actually Is
Okay, strip away all the Twitter discourse and what are you left with? A really simple file. Almost boringly simple. On purpose.
You’re publishing a plain-text Markdown document at your domain root. It hands AI models a curated index of your best content instead of making them reverse-engineer it from your HTML. Because your site is built for HUMANS. Menus, ads, popups, JavaScript. An AI trying to understand your docs has to download all that junk, strip it, and hope what’s left makes sense. Every wasted token costs money.
Instead? You give it a clean map. “Here are my 15 most important pages, here’s what each is about.” Done.
The spec is tiny:
- An H1 with your site or project name (the only required bit)
- A blockquote with a one-line summary of what the site does
- H2 sections with lists of Markdown links, each with a short description
Here’s what one actually looks like:
# Acme Widgets
> Acme Widgets sells industrial fasteners to manufacturers across the UK.
## Products
- [Hex bolts](https://example.com/hex-bolts): Our core product line, 40+ sizes
- [Custom orders](https://example.com/custom): Made-to-spec fasteners, 2-week lead time
## Support
- [Sizing guide](https://example.com/sizing): How to pick the right bolt
- [FAQ](https://example.com/faq): Shipping, returns, bulk pricingThat’s the whole format. The thinking behind it: documentation sites are built for humans, so an AI agent wanting your API reference has to download your HTML, strip the navigation, ads, and scripts, and hope the result still makes sense. Every wasted token costs money and latency. A curated Markdown map skips all that ten to twenty of your best links, described in plain language, instead of asking the model to guess from raw HTML.
Note what it is not: it doesn’t rank you, doesn’t index you, and doesn’t replace your sitemap. It’s a shortcut for a model that has already decided to read your site and wants to know where to look first.
llms.txt vs robots.txt: Don’t Confuse Them
The names invite confusion, so let’s kill it cleanly:
| robots.txt | llms.txt | |
|---|---|---|
| Job | Tells crawlers which parts of your site they may not fetch | Tells AI models which pages are worth reading first |
| Format | Plain text with crawler directives (Disallow: /admin/) | Markdown with headings and link lists |
| Enforcement | Followed by convention — well-behaved bots obey it | Enforces nothing; purely advisory |
| Status | Established web standard (Robots Exclusion Protocol) | Community proposal from 2024, v2 in 2026 |
| Do you need it? | Yes — every public site should have one | Optional; most valuable for docs, SaaS, and content-heavy sites |
| SEO effect | Direct — blocked pages don’t get indexed | None demonstrated; Google says it ignores the file |
They solve opposite problems. robots.txt is about access control — keeping bots out. llms.txt is about orientation — pointing bots to the good stuff. A site serious about AI visibility probably wants both. But if you only have time for one this week? Make it robots.txt with the AI crawlers explicitly allowed. That’s the one that actually matters.
The Honest 2026 Status: Does Anyone Actually Read It?
Okay, this is where most guides get vague and hand-wavy. I won’t. Let’s be specific.
No major AI provider confirms their crawlers fetch llms.txt. Not OpenAI, not Anthropic, not Perplexity, not Google. Some indie crawlers do. Dev-tool adoption is real September 2026 found 20 of 24 dev platforms serving a file. But the big answer engines? Silence.
Server logs are brutal. Ahrefs analyzed 137,000 domains: 97% of published files got ZERO requests in May 2026. Your file will likely sit unread — like a business card left after everyone’s gone home.
Google ignores it. Their docs say Search doesn’t use llms.txt, no effect on rankings. Anyone selling it as an SEO factor is selling something Google denied.
So is it useless? Not quite. It’s unproven. And there’s a difference. Standards sometimes matter before the big players admit it remember how long it took structured data to catch on? The file costs ten minutes and zero dollars. That’s a fair price for a lottery ticket on a standard that MIGHT matter later.
What’s NOT reasonable? Treating this file like it’s the thing that gets you cited — while your robots.txt is blocking GPTBot and your content is thin. That’s like polishing your business cards while your shop is on fire.
Should You Add One? A Decision Framework
Add it now if: you run docs, a SaaS product, or a content-heavy site. Or your audience already finds you through AI tools. Or you’ve done all the fundamentals and want the cheap insurance. Twenty minutes, zero downside. Go for it.
Do it later if: you’re a small business site still missing the basics. No Google Business Profile. No schema markup. Thin service pages. This file won’t save a site that AI can’t otherwise understand. Fix the foundation first — the file can wait.
Skip it (for now) if: your site is a single landing page. Or a portfolio. Or behind a login. There’s literally nothing to map. Don’t overthink it.
One more thing and this is important: don’t just dump your sitemap into it. The whole POINT is curation. Your 10 to 20 most important URLs, each with a genuine one-line description. An llms.txt that’s just your sitemap wearing a Markdown costume? Completely misses the point.
How to Create an llms.txt File
The process is genuinely simple ten minutes with coffee:
- Draft it. H1 with your site name, a blockquote summary, H2 sections (Products, Docs, Pricing, FAQ) with your best links and one-line descriptions.
- Curate ruthlessly. Ten to twenty links, not two hundred. If everything’s important, nothing is.
- Save as exactly
llms.txt— not.md, not.html. - Upload to domain root so it resolves at
yourdomain.com/llms.txt. WordPress: File Manager or FTP. Static host: output root. - Verify by visiting the URL. See Markdown as plain text? Done.
- Keep it fresh. Update when you launch key pages — a stale map is worse than none.
Anyone selling a four-figure “AI readiness package” around this file? Smile politely and walk away.
What Matters MORE for AI Visibility
If llms.txt is the garnish, here’s the actual meal. In rough order of impact:
1. Let the AI bots in. This is NUMBER ONE and it’s not close. Check your robots.txt right now — does it actually allow GPTBot, OAI-SearchBot (OpenAI), PerplexityBot, ClaudeBot, Google-Extended? Because SO many sites block AI crawlers by default. Or their security plugin does it without telling them. Then they wonder why ChatGPT never mentions them. This single check outweighs everything else in this article. Go do it.
2. Publish content that answers buying questions. AI answer engines cite pages that directly answer what the user asked. Comparisons. Pricing guides. How-tos. Honest pros-and-cons. Thin service pages that say “contact us for a quote”? The model has literally nothing to quote. Give it something.
3. Get mentioned where AI already looks. Models lean HARD on Reddit, YouTube, news sites, review platforms, marketplaces. One detailed Reddit thread about your product can outweigh months of on-site tweaking. I’m serious.
4. Add schema markup. Does it directly boost AI citations? Debated. But it’s cheap, it powers traditional rich results, and it removes ambiguity about what your pages are. Clean signals never hurt. Just do it.
5. Serve fast, server-rendered pages. AI crawlers are less patient than Googlebot. If your content needs JavaScript to render, check what a bot ACTUALLY sees fetch it with curl, not just DevTools. Slow or client-rendered pages get shallow crawls. Shallow crawls mean shallow citations.
6. Then drop in llms.txt. Low effort, possible upside, zero risk. The cherry. Not the cake. Never the cake.
Frequently Asked Questions
What is llms.txt?
A plain Markdown file at yourdomain.com/llms.txt containing a curated summary of your site and links to its most important pages. Jeremy Howard (Answer.AI) proposed it in September 2024, with v2 released in August 2026. It’s meant to help AI models understand a site’s structure without parsing raw HTML but no major AI provider has confirmed their crawlers read it.
Does Google use llms.txt?
No. Google states that Search ignores llms.txt entirely and that such files have no effect on rankings or visibility. If your goal is Google rankings or AI Overviews, your effort goes into content quality, schema, and crawlability — the file changes nothing there. Treat it as an AI-ecosystem extra, not a Google tactic.
Will llms.txt get my site cited by ChatGPT?
There’s no evidence it directly causes citations. ChatGPT cites pages its retrieval systems find relevant and trustworthy which comes down to crawl access, content quality, and being mentioned across the web. The file is a cheap, sensible addition for docs and SaaS sites, but it’s not a citation machine. Anyone promising otherwise is guessing.
Is llms.txt the same as robots.txt?
No — they do opposite jobs. robots.txt tells crawlers which parts of your site they may not fetch (access control, followed by convention). llms.txt suggests which pages are worth reading first (orientation, enforces nothing). You need robots.txt regardless; llms.txt is optional. Confusing the two is the most common mistake in this whole topic.
How do I add llms.txt to WordPress?
Upload a plain-text file named exactly llms.txt to your domain root via File Manager or FTP no plugin required. Alternatively, Rank Math can generate one for you. After uploading, visit yourdomain.com/llms.txt in a browser to confirm it loads as plain text. Keep it to 10–20 curated links with one-line descriptions, and update it when key pages change.
What should I put in my llms.txt file?
Your site name as H1, a one-line blockquote summary, then H2 sections with your 10–20 most important URLs — pricing, core product or service pages, documentation, key FAQs each with a genuine one-line description. Curate ruthlessly: the file’s value is telling AI what matters most, so a full sitemap dump defeats the purpose.
Final Thoughts
Here’s my take: llms.txt is a good idea wearing way too much hype. The file itself? Sound. A curated Markdown map costs nothing and might matter someday. But the 2026 data is clear almost nobody’s crawlers are reading it yet.
So add it in twenty minutes. Then spend your real energy where citations actually come from: letting AI bots crawl your site, publishing content worth quoting (see our AI content tools roundup), and building the kind of reputation that gets you mentioned.
The sites winning AI visibility in 2026? They’re not winning because of a text file. They’re winning because they’re genuinely the best answer. Be that.
About the Author
Asmara Khan is an SEO content writer and GEO specialist who covers AI search, freelancing, and online business for a UK/USA audience.
Figures checked October 2026. For deeper reading, see this honest llms.txt breakdown for business sites and the 24-vendor llms.txt check.
