Does your business website need an llms.txt file? An AI crawler readiness checklist

llms.txt is a markdown file at the root of a website that gives AI tools a short, curated description of the site and links to its most useful pages. It is a proposal, not a standard every engine has adopted, and Google's documentation for its AI features says you don't need to create new machine readable files or AI text files to appear in them. So an llms.txt file is cheap, harmless, and occasionally useful, and it matters far less than four checks underneath it: whether robots.txt lets the AI crawlers in, whether your sitemap and internal links reach every page that matters, whether each page states what it is about in plain text, and whether your structured data agrees with the visible page.

This guide covers all five, using the files this site publishes as the worked example. Nexus Multimedia, a San Diego agency, builds the same set into the sites it makes for law firms and businesses, generated from the pages rather than written by hand.

At a glance

• llms.txt. Optional, low cost, useful to tools that read it. Generate it, do not hand-write it.
• robots.txt. The control that matters most. Allow the search crawlers; decide on training crawlers separately.
• Sitemap and links. Every page that matters in the sitemap and linked from at least one other page.
• Plain-text answers. Each page says what it is about in its first paragraph, in text, not images.
• Structured data. Markup that matches what the page shows, and nothing it does not.

What is llms.txt, and where did it come from?

It was proposed by Jeremy Howard on September 3, 2024, at llmstxt.org, as "A proposal to standardise on using an /llms.txt file to provide information to help agents use a website." The format is plain markdown. The only required part is "An H1 with the name of the project or site." After it come a blockquote summarising the site, optional notes, and H2 sections containing lists of links, each with a name and an optional description. A section headed "Optional" is used, by convention, for secondary material an agent can skip.

The proposal also suggests that sites publish a clean markdown version of each page at the same address with .md added. A file called llms-full.txt, with the full text of the site in one document, is a related convention some sites follow; it is not part of the proposal itself.

Which AI systems read llms.txt?

Fewer than the attention suggests, going by what the companies document. Google is the clearest: its documentation on AI features and your website says "You don't need to create new machine readable files, AI text files, or markup to appear in these features." The guidance OpenAI, Anthropic, and Perplexity publish for site owners is about robots.txt: which crawlers they run, what each one is for, and how to allow or block it.

The proposal's own site notes that AI labs publish llms.txt files for their developer documentation. That tells you the format is useful for docs an agent is working through. It does not tell you an assistant reads yours before recommending a business. Treat the file as a courtesy to tools that use it, and put your effort into the checks below.

What should an llms.txt file contain?

A name, a one-line description, and the pages you would want an assistant to read first, grouped the way a visitor would look for them. Ours, at /llms.txt, is laid out like this (entries shortened):

# Nexus Multimedia
> AI Search Visibility and Google SEO for law firms and businesses.

## Services
- [Google SEO for Law Firms and Practices](https://nexusmultimedia.ai/services/seo/): Rank for the searches that bring in cases.
...
## Industries
## Locations
## Blog
## Company

Each entry takes its title and description from the page itself, so the file says what the page says. It is generated by a script every time the site changes, alongside /llms-full.txt with the full text of every page. A hand-written llms.txt drifts out of date as soon as the site changes, and an out-of-date map is worse than none.

Does your robots.txt let the AI crawlers in?

This is the check that decides whether an assistant can use your pages at all. Google's robots.txt guide describes the file as "used primarily to manage crawler traffic to your site," and notes that it "is not a mechanism for keeping a web page out of Google." Each AI company runs more than one crawler, and the difference between them matters.

OpenAI

• OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features." OpenAI's crawler documentation says "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers."
• GPTBot crawls content that may be used in training; disallowing it "indicates a site's content should not be used in training generative AI foundation models."
• ChatGPT-User handles actions a user asks for, and "Because these actions are initiated by a user, robots.txt rules may not apply."

Anthropic

• ClaudeBot collects web content that could contribute to model training. Blocking it signals that future material should be excluded from training.
• Claude-User fetches pages when a person asks Claude a question; blocking it may reduce visibility for user-directed web search.
• Claude-SearchBot "navigates the web to improve search result quality for users"; blocking it may reduce visibility and accuracy in user search results. Anthropic's help center says its bots honor robots.txt.

Perplexity

• PerplexityBot "is designed to surface and link websites in search results on Perplexity" and is "not used to crawl content for AI foundation models," according to Perplexity's crawler documentation.
• Perplexity-User supports user actions, and "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

Google

• Googlebot is the control for Google Search, including its AI features; Google's AI features documentation says robots.txt directives for Googlebot are how site owners manage access.
• Google-Extended is a robots.txt token, not a separate crawler, that manages whether content is used for training Gemini models and for grounding. Google's crawler documentation says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

The practical split is between search crawlers and training crawlers. OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot decide whether you can be found and recommended. GPTBot and ClaudeBot decide whether your content is used to train models. Google-Extended covers both training Gemini models and grounding Gemini's answers, so blocking it can keep you out of Gemini's grounded answers as well. OpenAI says "Each setting is independent of the others," so a business can allow search and decline training. A business that wants to be recommended should not block the search crawlers. Ours allows every crawler through the catch-all rule and also names the main AI crawlers. This is the live robots.txt, in full:

# Nexus Multimedia robots.txt
# We optimize for both Google and AI search. Crawlers that train and retrieve
# for ChatGPT, Claude, Perplexity, Google AI Overviews, and Microsoft Copilot
# are explicitly welcomed.

User-agent: *
Allow: /
Disallow: /tmp/
Disallow: /home-
Disallow: /assets/logos/

# AI and answer-engine crawlers, same access as everyone else.
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: CCBot
User-agent: Bingbot
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
Allow: /
Disallow: /tmp/
Disallow: /home-
Disallow: /assets/logos/

Sitemap: https://nexusmultimedia.ai/sitemap.xml

# AI crawler discovery. These are not standard robots.txt directives and no
# crawler parses them, so they are kept as comments. The files stay discoverable
# at their well-known paths.
# Llms: https://nexusmultimedia.ai/llms.txt
# Llms-full: https://nexusmultimedia.ai/llms-full.txt

The Disallow lines ask every crawler to skip a few private paths. The llms files are mentioned only in the closing comments, because they are not robots.txt directives and no crawler parses them as such.

Can crawlers reach every page that matters?

A page nobody links to is a page a crawler may never find. Every page you want an assistant to know about should be in the sitemap and linked from at least one other page in the body copy, not only from a footer. Check for orphans, broken internal links, and pages the sitemap lists that no longer exist. How to structure service and location pages so each has a clear job is covered in practice area page structure, and the principle applies to any business.

Does each page say what it is about in plain text?

An assistant can only quote what it can read. Put the answer to the page's main question in its first paragraph, in text. Phrase headings the way people ask questions. Keep important information out of images, and make sure the content is in the HTML rather than loaded only by scripts after the page appears. If a page has a FAQ, make the questions the real ones customers ask, and keep the answers short enough to stand alone.

Does your structured data match the page?

Google's AI features documentation lists "Making sure your structured data matches the visible text on the page" among the practices that support appearing in those features. The useful set is small: one Organization node with consistent details, Service nodes for what you sell, BlogPosting on articles with real dates, BreadcrumbList for the site's shape, and FAQPage only where a visible FAQ exists, mirroring it word for word. Markup that describes something a visitor cannot see is a contradiction, and contradictions are what get a page set aside. The longer treatment, with examples, is in schema markup for personal injury law firms.

Are markdown versions of your pages worth publishing?

Sometimes. This site publishes a markdown twin of every page, generated from the HTML with the navigation and repeated calls to action stripped out, so a tool that prefers markdown gets the content without the chrome. That follows the llms.txt proposal's suggestion of a clean .md version at the same address. It is worth doing when it can be generated automatically, as ours is. It is not worth maintaining by hand, and it does nothing for a page that fails the checks above.

A 12-point AI crawler readiness checklist

1. robots.txt allows OAI-SearchBot, PerplexityBot, Claude-SearchBot, and Googlebot.
2. The training crawlers (GPTBot, ClaudeBot) are allowed or blocked as a deliberate decision, and Google-Extended is decided with Gemini's grounded answers in mind.
3. robots.txt points to the sitemap.
4. The sitemap lists every page that matters, and nothing that no longer exists.
5. Every page has at least one link to it from the body of another page.
6. No broken internal links.
7. Each page answers its main question in its first paragraph, in text.
8. Headings read as the questions customers ask.
9. The content is in the HTML, not only rendered by scripts.
10. Structured data matches the visible page, with FAQPage only where a visible FAQ exists.
11. An llms.txt file, generated from the site, if you publish one.
12. The same business name, address, and phone number on the site, the Google Business Profile, and every directory.

How we handle this

The sites we build ship with a robots.txt that names the AI crawlers, a sitemap kept in step with the pages, an llms.txt and llms-full.txt, and a markdown twin of each page, all generated rather than hand-edited. That is part of our website design work and the technical side of our AI search visibility program; the services overview shows the rest. The files are the floor, not the ceiling: being recommended still depends on the content and sources covered in how to get your business recommended by AI, and the vocabulary around all of it is untangled in GEO, AEO, and AI search visibility.

Frequently asked questions