A new standard is emerging that tells AI tools exactly what's on your website and what to prioritize. Most businesses have never heard of it. Here's what it is and how to add it.
A file most websites don't have — but should
There is a small text file that an increasing number of AI tools check before crawling a website. It lives at the root of your domain, it takes about 30 minutes to create, and it can meaningfully influence how AI search engines understand and cite your business. Most businesses have never heard of it.
It's called llms.txt.
If that sounds technical, it is — but the concept is simple, and the implications are significant enough that any business serious about AI visibility should understand it.
What llms.txt actually is
llms.txt is a plain text file placed at the root of your website (e.g., https://yourdomain.com/llms.txt) that provides a machine-readable summary of your site's content, purpose, and structure. It tells AI language models — the systems powering ChatGPT, Perplexity, Claude, and similar tools — what your site is about, what content it contains, and how that content is organized.
Think of it as a sitemap, but written for AI rather than for search engine crawlers. A sitemap tells Google what pages exist on your site. llms.txt tells AI language models what those pages mean, what topics they cover, and which content should be prioritized when the AI is building its understanding of who you are and what you do.
The standard was proposed by Jeremy Howard in 2024 and has been adopted by a growing number of technology companies, publishers, and forward-thinking businesses. It's not an official W3C standard yet, but major AI labs and crawlers have acknowledged it, and some explicitly check for it.
Why it matters for your business
When an AI tool like Perplexity or ChatGPT (with web browsing) crawls your site, it's making decisions about what to read, how much weight to give different pages, and how to summarize your business to a user who asked a relevant question. Without llms.txt, it's guessing based on your page structure, headers, and meta descriptions — the same signals it uses for everything else.
With llms.txt, you're providing explicit guidance. You're saying: here is what this site is, here are the most important pages, here is the content that represents us best. That guidance can influence which pages get read more carefully, which content is treated as authoritative, and ultimately, whether your business is cited accurately.
The analogy is a press kit. When a journalist covers your business, a good press kit doesn't make them write a positive story — but it gives them the right context, the right quotes, and the right facts. It makes accuracy more likely. llms.txt does the same thing for AI tools.
What goes in an llms.txt file
An effective llms.txt file has several sections:
A brief description of the business. One to three sentences that clearly explain what the organization is, what it does, and who it serves. This is the context an AI uses to interpret everything else on the site.
A list of key pages with descriptions. The most important pages on your site — services, about, case studies — listed with a one-sentence description of each. This tells AI crawlers which pages matter most and what they contain.
A content guide. A high-level map of the site's content categories: what topics you cover, what types of content exist (articles, guides, case studies), and what subject matter your site is authoritative on.
Optional: usage guidance. Some llms.txt files include guidance about how AI tools should use the content — for example, clarifying that certain content is for informational purposes only, or noting that pricing information should be confirmed directly.
The file should be human-readable and concise. It's not a data dump of every URL on your site — that's what your sitemap.xml is for. llms.txt is a curated, context-rich guide.
How llms.txt differs from robots.txt and sitemap.xml
These three files serve distinct purposes and work together:
robots.txt tells crawlers what they're allowed to access. It's about permission — which parts of your site should and shouldn't be crawled. An AI crawler that sees robots.txt respects it as a directive.
sitemap.xml tells crawlers what pages exist on your site. It's a comprehensive list of URLs to help ensure full indexation. It says nothing about what those pages mean or how important they are relative to each other.
llms.txt tells AI systems what your content means and how to prioritize it. It's about comprehension and context, not permission or discovery. It helps AI tools understand your business accurately before they cite it.
A complete AI visibility setup uses all three.
An extended version: llms-full.txt
Some site owners also publish an llms-full.txt file at their domain root. Where llms.txt is a concise summary, llms-full.txt contains the complete text of your most important pages — the actual content, formatted for AI consumption, without HTML tags and navigation clutter.
This is particularly useful because AI crawlers sometimes have difficulty extracting clean text from complex web pages with heavy JavaScript, dynamic content, or cluttered layouts. llms-full.txt gives them the content directly, in the cleanest possible format. For content-heavy sites — publishers, resource centers, agencies with extensive guides — llms-full.txt can meaningfully improve how comprehensively AI tools represent your content.
How to create your llms.txt file
Creating llms.txt doesn't require a developer, though a developer will help you get it right. The file is plain text. You can write it in a text editor and upload it to your web server root.
The structure looks like this:
```
# [Your Business Name]
> [One to two sentence description of your business, what you do, and who you serve.]
Key Pages
- [Home](https://yourdomain.com/): [One-sentence description]
- [Services](https://yourdomain.com/services/): [One-sentence description]
- [About](https://yourdomain.com/about/): [One-sentence description]
Content
- [Resource title](https://yourdomain.com/resources/slug/): [One-sentence description]
Topics Covered
[Brief description of the subject matter your site is authoritative on]
```
For a Next.js or similar web application, the file can also be served dynamically from a route handler — which allows it to stay automatically in sync as content on your site changes.
The early-mover advantage is real
llms.txt is new enough that the majority of websites — including most of your competitors — don't have one. AI tools are actively developing their standards for how to interpret and use these files. The businesses that establish well-structured llms.txt files now will have their content formats embedded in how AI tools understand their market before competitors catch up.
This is not a theoretical advantage. AI systems that have crawled and understood a business's llms.txt file are more likely to accurately represent that business in answers, more likely to cite specific pages from that business, and more likely to return that business as a recommendation when a user asks a directly relevant question.
The cost of adding llms.txt to your site is low. The potential upside — accurate, consistent representation in the AI answers your potential customers are seeing every day — is significant.
For a broader look at what AI visibility requires, see our guide on making your website visible to AI search engines. And if you want to understand the technical signals AI tools weight most heavily, our guide on schema markup for AI search goes deeper on structured data.
Ready to put this into action?
We'll audit your digital presence and show you exactly where your biggest opportunities are — free, no commitment.