ClaudeBot, Claude-SearchBot and Claude-User explained

Anthropic runs three web crawlers with distinct jobs: ClaudeBot collects web content that could contribute to model training, Claude-SearchBot navigates the web to improve search result quality, and Claude-User may fetch pages when individuals ask Claude questions. Because Anthropic splits training, search indexing and user requests across three separate user agents, you can block model training in your robots.txt file without cutting your site out of Claude's search answers or live user queries.

The three Anthropic bots and what each one does

Anthropic separates its web activity into three user agents, much like OpenAI splits training, search and user actions across GPTBot, OAI-SearchBot and ChatGPT-User. Each bot has a specific role, and blocking one does not block the others unless you tell your robots.txt to disallow all three.

Here is how Anthropic defines the three bots in its official crawler documentation:

• ClaudeBot is the training crawler. Disallowing it tells Anthropic that your future content shouldn't go into model training datasets.
• Claude-SearchBot is the search crawler. If you block it, Anthropic won't index your pages for search, which can reduce how often your site shows up when Claude answers search-backed prompts.
• Claude-User may access a page when a person asks Claude a question. Blocking it stops Claude from retrieving your pages for those questions.

How Anthropic's bots handle robots.txt and IP lists

All three Anthropic crawlers obey standard robots.txt directives, and unlike Applebot, they also support the non-standard Crawl-delay extension. If ClaudeBot is hitting your server too fast, you can slow it down without blocking it completely.

Anthropic notes two important rules about how its opt-outs work:

1. You must set rules per subdomain. A robots.txt file on example.com does not cover blog.example.com or docs.example.com. Each subdomain that serves traffic needs its own robots.txt directives.
2. Blocking IP addresses isn't a reliable opt-out. Anthropic warns that IP blocks may not work correctly or persistently guarantee an opt-out. Use robots.txt instead.

Anthropic publishes the IP addresses its crawlers use at https://claude.com/crawling/bots.json and says a source IP on that list indicates the crawler is coming from Anthropic. Clicktag doesn't check that list: as covered in our AI crawler docs, Claude's bots count as verified when Cloudflare verifies them.

robots.txt examples for ClaudeBot, Claude-SearchBot and Claude-User

If you want to block AI crawlers for training but still show up when someone asks Claude a question, block ClaudeBot while allowing search indexing (Claude-SearchBot) and user fetches (Claude-User). Put this in /robots.txt at the root of each subdomain:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

If you don't mind ClaudeBot reading your site but want to slow it down, Anthropic supports Crawl-delay:

User-agent: ClaudeBot
Crawl-delay: 1

And if you want to keep Anthropic off your site completely, disallow all three user agents:

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /

How to see Claude's bots in Clicktag

Browser analytics scripts never see ClaudeBot, because it fetches raw HTML and doesn't run JavaScript. To show you AI crawler analytics without adding code to your site, Clicktag reads crawler requests from Cloudflare's analytics using a read-only API token.

If your site is verified in Clicktag and proxied through Cloudflare (any plan works, including Free), setup takes three steps:

1. Open your site's settings in Clicktag, go to Integrations, and find Cloudflare.
2. Click Create a read-only token. Cloudflare opens with Zone Read and Analytics Read pre-selected for your zones. Create the token and copy it.
3. Paste the token into Paste the token and click Connect.

Clicktag imports your last 30 days of crawler history on connect and then syncs every 15 minutes. Because Cloudflare drops its own analytics after 31 days, Clicktag stores your hourly crawler counts so your history keeps growing over time.

Because anyone can type ClaudeBot into a user agent string, Clicktag separates verified hits from unverified ones on your Crawlers tab.

For OpenAI, Perplexity, Google, Microsoft, Apple, Common Crawl and DuckDuckGo, Clicktag checks both Cloudflare's verification and the company's published IP ranges. For Anthropic, ClaudeBot, Claude-SearchBot and Claude-User count as verified when Cloudflare verifies them in its AI Crawler, AI Search or AI Assistant categories. Requests that claim to be a Claude bot without passing Cloudflare verification are kept separate in the unverified count, and verified requests that your server or firewall turned away with a 401, 403 or 429 status code show up in the blocked count.

Crawls versus visits: when people click through from Claude

A hit from Claude-SearchBot or Claude-User is a server fetch, not a person visiting your website. Those requests stay on the Crawlers tab and are never mixed into your visitor or pageview counts.

When a real person reads an answer in Claude, clicks a citation link, and lands on your site in their browser, our latest.js script (about 4 KB) runs and records a normal visit. In your AI traffic analytics and traffic sources, Claude shows up as its own source alongside ChatGPT, Perplexity, Gemini and Copilot (unless the app sends no referrer and no UTM parameters, in which case the visit lands in "Direct / none").

You can also create goals (up to 50 goals and funnels per site), which apply to your existing data too, and break them down by source to see how visitors coming from Claude convert compared to visitors from google.com or chatgpt.com.

FAQ

Does Clicktag block ClaudeBot for me?

No. Clicktag reports crawler activity and shows how many verified hits were answered with 401, 403 or 429 status codes, but it doesn't block bots. To block ClaudeBot, update your /robots.txt file or configure Cloudflare's AI Crawl Control in your Cloudflare dashboard.

Why do I see unverified ClaudeBot hits in Clicktag?

Anyone can run a script or scraper with ClaudeBot in the User-Agent header. Clicktag only counts a ClaudeBot, Claude-SearchBot or Claude-User request as verified when Cloudflare verifies the bot, and puts the rest under unverified so fake bots don't skew your numbers.

Do subdomains share the same robots.txt and Clicktag crawler reports?

No on both counts. Anthropic requires a separate robots.txt opt-out on each subdomain, and Clicktag's crawler tracking for a site like example.com only counts requests to example.com and www.example.com (though one Cloudflare token in Clicktag can cover up to 50 sites in your account).

Which requests count as pages crawled on the Crawlers tab?

Pages crawled counts the distinct paths with at least one verified hit. Text files like /robots.txt and /llms.txt count; scripts, styles, fonts, images and Cloudflare's /cdn-cgi/ endpoints don't, so asset loads don't inflate your numbers. The Pages list shows the top 100 paths.

What isn't included in Clicktag?

Clicktag doesn't store IP addresses (they're checked during sync and discarded), and it doesn't offer session recordings, heatmaps, bounce rate, revenue tracking or Google Analytics imports. It is a hosted tool (not open source or self-hostable) that requires a Google account to sign in, and you can inspect a synthetic sample dashboard first at the live demo.