AI crawler list: every AI bot user agent and what it does

Most AI bots fall into three groups: training crawlers that collect pages for model training, search crawlers that index pages for AI answers, and user fetchers that visit a page because of a user's action, such as a question in a chat. A few companies also publish control tokens that only go inside robots.txt to control how crawled content is used, without running as a separate bot.

AI crawler list and user agent tokens

Here are 17 main AI crawler tokens, who runs them, what they do, and how they handle robots.txt rules according to each company's documentation.

A few vendors also run extra bots alongside the ones in the table. Amazon separates search and live Alexa fetches into Amzn-SearchBot and Amzn-User. Mistral AI uses MistralAI-Index for search and MistralAI-Training for training datasets. Clicktag also tracks Microsoft's Bingbot as a search engine crawler.

Bytespider, Meta-ExternalAgent and Meta-ExternalFetcher

You may also see Bytespider (ByteDance), Meta-ExternalAgent (Meta), and Meta-ExternalFetcher (Meta) in server logs and on Clicktag's Crawlers tab.

Meta's documentation describes Meta-ExternalAgent as crawling for use cases such as training foundation AI models or improving products by indexing content directly, and Meta-ExternalFetcher as fetching individual links at a user's request. We found no official ByteDance documentation for Bytespider. Because their rules are unverified or undocumented, we list them separately without claims about how they treat robots.txt.

The three kinds of AI bots explained

Not every bot from OpenAI, Anthropic, or Google wants the same thing from your server. Grouping them by what they do makes it much easier to decide which ones to allow.

• Training crawlers and control tokens: Bots like GPTBot, ClaudeBot and Amazonbot collect pages that may be used to train AI models, and CCBot collects pages for Common Crawl's open repository of web crawl data. Google-Extended and Applebot-Extended do not crawl on their own; they are control tokens you put in robots.txt to tell Google and Apple not to train models on the pages Googlebot and Applebot fetch.
• Search crawlers: Bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so your site can show up in AI search answers, and DuckAssistBot crawls pages in real time for DuckDuckGo's AI-assisted answers. Blocking OAI-SearchBot keeps your pages out of ChatGPT search answers (though navigational links can still appear), while blocking Google-Extended has no effect on your Google Search inclusion or ranking.
• User fetchers: Tokens like ChatGPT-User, Claude-User, Perplexity-User, and MistralAI-User visit a page because of a user's action in an AI assistant, such as asking a question. OpenAI notes that robots.txt rules may not apply to ChatGPT-User, and Perplexity states that Perplexity-User generally ignores robots.txt rules.

Blocking training crawlers tells AI companies not to train future models on your pages, while leaving search crawlers open lets AI search engines show and link to your site.

A robots.txt template that blocks training and keeps search

If you want your site to show up in AI search answers and regular search results while opting out of the training crawlers and control tokens in the table, add these rules to your /robots.txt file.

# Block AI training crawlers and training control tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /

# Allow AI search crawlers and traditional search engines
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckAssistBot
Allow: /

Anthropic's crawlers require rules to be set on each subdomain separately, and they support Crawl-delay if you want to slow them down instead of blocking them. OpenAI says robots.txt changes take about 24 hours to apply and Perplexity says up to 24 hours, while DuckDuckGo says a DuckAssistBot opt-out takes effect after 72 hours.

Why JavaScript analytics miss AI crawlers

JavaScript trackers only fire when a browser loads a page and runs scripts. A Vercel and MERJ study published in December 2024 found that GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and PerplexityBot fetch raw HTML without rendering JavaScript, so client-side analytics scripts never see them. Even when a search crawler does run JavaScript, Clicktag's latest.js script (about 4 KB) recognizes it as a bot by its user agent and leaves it out of your human visitor numbers.

When a person clicks a cited link inside ChatGPT or Perplexity, that visit shows up in your traffic sources (ChatGPT adds utm_source=chatgpt.com to its links). To see the bots themselves fetching your HTML, you have to read the requests at the network edge.

How to see which AI bots hit your site with Clicktag

Clicktag tracks AI crawlers per page by reading your Cloudflare request analytics through a read-only API token. It works on any Cloudflare plan, including Cloudflare Free, and requires no code changes on your site.

1. Verify your website in Clicktag (with the latest.js tag on your homepage, a DNS TXT record, or a file in /.well-known/) and make sure the domain is proxied through Cloudflare.
2. Open your site settings in Clicktag, go to Integrations, and find Cloudflare.
3. Click Create a read-only token. Cloudflare opens with Zone Read and Analytics Read pre-selected for your zones. Create the token and copy it.
4. Paste the token into Paste the token and click Connect.

On connect, Clicktag imports your last 30 days of crawler history a week at a time, then syncs every 15 minutes. While Cloudflare drops this data after 31 days, Clicktag stores the hourly counts so your crawler history keeps growing over time.

On the Crawlers tab, Clicktag keeps verified and unverified hits apart. Anyone can fake a user agent string, so a request only counts as verified if Cloudflare verified it in its AI Crawler, AI Search, AI Assistant or Search Engine Crawler category, or if it came from an IP range published by OpenAI, Perplexity, Google, Microsoft, Apple, Common Crawl, or DuckDuckGo. IP addresses are checked during the 15-minute sync and are never stored.

The Pages list shows the top 100 paths, including /robots.txt and /llms.txt, with how many different bots read each one. The Bots panel shows how many of each bot's verified hits your site answered with a 401, 403, or 429 status code. Click a bot or a page to filter the whole tab.

FAQ

Does Clicktag block AI crawlers?

No, Clicktag is a reporting tool that shows you which verified and unverified bots hit your pages and what HTTP status codes your site returned. To block them, you edit your /robots.txt file or configure bot rules in Cloudflare.

Do I need a paid Cloudflare or Clicktag plan to track AI crawlers?

No. Clicktag is free with no credit card required for up to 50 websites per account, and the Cloudflare integration works on any Cloudflare plan, including Free.

How does Clicktag spot fake GPTBot or Googlebot requests?

Clicktag checks whether Cloudflare verified the bot or whether the request IP matches the official IP ranges published by OpenAI, Perplexity, Google, Microsoft, Apple, Common Crawl, and DuckDuckGo. Requests that claim a bot name in their user agent without passing either check are counted as unverified, and IP addresses are never stored.

Why are my ClaudeBot hits only verified through Cloudflare?

Anthropic publishes the IP addresses its crawlers use at https://claude.com/crawling/bots.json, but Clicktag doesn't check that list. ClaudeBot, Claude-SearchBot, and Claude-User count as verified when Cloudflare verifies them.

Which requests count toward pages crawled on the Crawlers tab?

Pages crawled counts the distinct paths with at least one verified hit. Only requests to your site's hostname and its www twin count, including text files like /robots.txt and /llms.txt. Static assets that a crawler loads with a page (scripts, styles, fonts, and images), Cloudflare /cdn-cgi/ endpoints, and other subdomains in the same zone are excluded, and the Bots and Pages panels each show the top 100 entries.