llms.txt: what it is and whether AI crawlers read it

llms.txt is a proposed Markdown file placed at /llms.txt to give language models a short map of a website. Jeremy Howard published the proposal on September 3, 2024. Despite the interest around llms.txt generators, no official crawler documentation from Google, OpenAI or Anthropic states that their search or training bots read /llms.txt. They use robots.txt for crawler control and index standard web pages.

What is the llms.txt proposal?

Published at llmstxt.org on September 3, 2024 (the page was last modified on August 10, 2026), the specification proposes a standard /llms.txt file at the root of a site, or under a sub-path, to help AI agents use a website without parsing complex HTML menus.

The format is plain Markdown with a set structure:

• One H1 heading with the name of the site or project.
• A blockquote (>) with a short summary.
• Zero or more paragraphs of extra detail.
• H2 sections containing Markdown lists of links, with an optional colon and note after each link.

Here is a minimal example that follows the llmstxt.org format:

# Example App

> Example App is a project management tool for small engineering teams.

Use these links to read our documentation and API reference.

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): Set up a workspace
- [API reference](https://example.com/docs/api.md): Endpoints and authentication

## Optional

- [Changelog](https://example.com/changelog.md): Recent product updates

The proposal also suggests offering clean Markdown versions of individual pages by appending .md to the URL (such as page.html.md or page.md), and linking to them in HTML with rel="alternate" type="text/markdown" or rel="describedby".

Does llms.txt work for AI search?

No official documentation from OpenAI, Anthropic, Google or Perplexity says their crawlers look for /llms.txt when indexing the web or answering questions.

Publishing /llms.txt is not a documented way to get indexed or cited by ChatGPT, Claude, Perplexity or Google AI Overviews.

The llmstxt.org site points out that OpenAI, Anthropic and Gemini publish llms.txt files for their own developer documentation. That covers their own documentation, and it is not a statement that their web crawlers or answer engines read /llms.txt on your site.

Here is how /llms.txt compares with the files crawlers actually document:

Google's documentation on AI features and your website states: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."

What actually matters for AI crawlers

Instead of generating an /llms.txt file and hoping bots look for it, check two things that vendor docs and crawler measurements back up.

First, put your content in the HTML the server sends. A study by Vercel and MERJ published on December 17, 2024 found that "none of the major AI crawlers currently render JavaScript", covering OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. Gemini uses Googlebot infrastructure with full rendering, and Applebot renders JavaScript, while CCBot does not. If your text only appears after client-side JavaScript runs, crawlers that don't render JavaScript never see it.

Second, allow the search and user-action bots in your robots.txt file. Each engine documents which bot powers its answers:

• ChatGPT search: OAI-SearchBot surfaces websites in ChatGPT search features. OpenAI's documentation notes that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and robots.txt updates take about 24 hours to adjust. OAI-SearchBot and GPTBot (used for model training) have independent settings, so you can block GPTBot while allowing OAI-SearchBot.
• Perplexity: PerplexityBot surfaces and links websites in Perplexity search results and is not used to crawl content for AI foundation models. Perplexity-User fetches pages when a user asks a question.
• Claude: Claude-SearchBot indexes content for search and Claude-User retrieves pages for user queries, while ClaudeBot collects content for model training.
• Google AI Overviews and AI Mode: These run on Google Search. A page must be indexed and eligible to show in Google Search with a snippet, using normal Googlebot rules.

You can read more about how these bots work on our AI SEO and AI crawler list pages.

How to check if bots read your /llms.txt file

You don't have to guess whether any bot requests /llms.txt on your domain. Script-based analytics can't count these requests: /llms.txt is a plain text file that runs no JavaScript, and GPTBot, ClaudeBot and PerplexityBot never run JavaScript on your HTML pages either.

Clicktag's AI crawler analytics reads crawler requests directly from Cloudflare using a read-only token, and text files like /robots.txt and /llms.txt count as pages in your reports.

If your domain is proxied through Cloudflare (any plan, including Free), you can set it up in three steps:

1. Verify your site in Clicktag, open your site's settings, go to Integrations and find Cloudflare.
2. Click Create a read-only token. Cloudflare opens its token page with Zone Read and Analytics Read already selected. Create the token and copy it.
3. Paste it into Paste the token and click Connect.

Clicktag imports the last 30 days of crawler history on connect and syncs every 15 minutes after that, with no code needed on your site.

Open the Crawlers tab on your dashboard and look at the Pages panel, which lists the top 100 paths with how many different bots read each one. Click /llms.txt (if it appears) to filter the whole tab and see which bots requested it, how many hits were verified, and what HTTP status codes your server returned. The full setup is in the AI crawlers documentation.

How Clicktag verifies AI crawlers

Anyone can send a request to /llms.txt with GPTBot or ClaudeBot in the user agent string. Clicktag separates verified hits from unverified claims so a scraper or a curious developer testing a tool doesn't look like OpenAI in your stats.

A hit counts as verified only when Cloudflare verified the bot in its AI Crawler, AI Search, AI Assistant or Search Engine Crawler category, or when the request came from an IP range published by OpenAI, Perplexity, Google, Microsoft, Apple, Common Crawl or DuckDuckGo. IP addresses are checked during the sync and are never stored.

If you also install Clicktag's latest.js script (about 4 KB) on your site, you can see human visits from AI answer engines alongside your crawler numbers. ChatGPT adds utm_source=chatgpt.com to links it shows, and Clicktag reads utm_source first, so those visits show up as chatgpt.com in your Sources report.

FAQ

Does Google or OpenAI use llms.txt for search?

No official crawler documentation from Google, OpenAI, Anthropic or Perplexity states that they read /llms.txt. Google states that there are no special optimizations or extra requirements needed to appear in AI Overviews or AI Mode beyond standard Google Search indexing.

Does publishing an llms.txt file hurt my site?

There is no documented downside: it is a static Markdown file and it doesn't change how crawlers read /robots.txt. It also has no documented support from major search and AI crawlers, so measure whether anything fetches it before spending time maintaining it.

Does Clicktag count crawler hits to /llms.txt and /robots.txt?

Yes. While Clicktag ignores scripts, styles, fonts, images and Cloudflare's /cdn-cgi/ endpoints in crawler reports, text files like /llms.txt and /robots.txt count as pages and appear in the top 100 paths on the Crawlers tab if verified bots request them.

What do I need to track AI crawlers in Clicktag?

Your site must be verified in Clicktag and its domain must be proxied through Cloudflare on any Cloudflare plan, including the Free plan. You only connect a read-only Cloudflare token with Zone Read and Analytics Read permissions; you don't need to install any code on your site for crawler tracking.

Can I block AI crawlers from Clicktag?

No. Clicktag shows crawler activity, including a blocked count of verified hits that your site answered with 401, 403 or 429, so you can see if your firewall or bot rules are turning bots away, but it does not block bots itself.

What are Clicktag's limits?

Clicktag is free, needs no credit card and covers up to 50 websites per account, with Google sign-in. It does not offer session recordings, heatmaps, bounce rate, A/B tests, revenue tracking or Google Analytics imports.