Bot traffic: how to tell crawlers from real visitors

AI crawlers like GPTBot, ClaudeBot and PerplexityBot never show up in JavaScript web analytics, because they fetch raw HTML and do not run client-side code. Search engine crawlers that do run JavaScript identify themselves in their user agent string, so analytics tools can drop them. To see what crawlers are actually doing on your site without polluting your visitor numbers, you need a tracker that filters out bots plus request data from the network edge, such as Cloudflare.

The main kinds of automated traffic

Not all automated requests hit your site for the same reason. Most fall into a few groups:

• Search engine crawlers: Bots like Googlebot, Bingbot, DuckDuckBot and Baiduspider index pages for traditional search results.
• AI training crawlers: Bots like GPTBot (OpenAI), ClaudeBot (Anthropic), Amazonbot, CCBot (Common Crawl) and meta-externalagent collect web content that can be used to train AI models.
• AI search crawlers: Bots like OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and ExaSearchBot index pages specifically for AI answers and search features.
• User-triggered AI fetchers: When someone asks an AI assistant a question, bots like ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot, MistralAI-User, Amzn-User and meta-externalfetcher can fetch pages in real time to answer it.
• SEO tools, link previews and webhooks: Commercial crawlers, social media preview fetchers, uptime checks, advertising bots and security scanners.
• Unverified scrapers: Scripts that fake a browser user agent or pretend to be a known bot like Googlebot from an unauthorized IP address.

Why most bots never show in JavaScript analytics

Standard website visitor tracking relies on a script tag in the browser. Clicktag uses one async latest.js script that is about 4 KB compressed.

When GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot or PerplexityBot visit a URL, they download the raw HTML document and stop there. They never run JavaScript, so latest.js never fires and those hits never reach your visitor reports.

Some search engine crawlers do render pages and run JavaScript. When they do, Clicktag recognizes them as bots from their user agent and leaves them out of your visits and pageviews. Hits detected as bots are kept out of every report and the live view, and visitors are counted with a daily salted SHA-256 hash, with no cookies and no stored IP addresses.

How bot verification works

Anyone can write a script and set its user agent header to claim it is GPTBot/1.4 or Googlebot. Relying on the user agent string alone gives you fake data.

Because Cloudflare sits in front of your site and proxies every request, it sees both the user agent and the network origin. When you use AI crawler analytics in Clicktag, a request only counts as Verified if it meets one of two checks:

1. Cloudflare verification: Cloudflare verified the bot in its AI Crawler, AI Search, AI Assistant or Search Engine Crawler category. Clicktag's IP check doesn't cover Anthropic, so ClaudeBot, Claude-SearchBot and Claude-User count as verified only when Cloudflare verifies them.
2. Published IP ranges: The request came from an IP range published by the bot's operator. Clicktag checks the official IP lists of OpenAI, Perplexity, Google, Microsoft, Apple, Common Crawl and DuckDuckGo. Common Crawl also uses dedicated IP ranges with reverse DNS under crawl.commoncrawl.org.

Checking published IPs catches real bots that Cloudflare misses. For example, Cloudflare no longer verifies PerplexityBot, but on our own sites about 90% of the PerplexityBot hits Cloudflare did not verify came directly from Perplexity's published IP ranges. Clicktag checks the IP during the sync and never stores IP addresses.

A request that claims to be a known bot like GPTBot or Googlebot without passing Cloudflare verification or an official IP check is separated into the unverified count.

On our own sites, unverified hits claiming to be OAI-SearchBot, GPTBot, ChatGPT-User or Googlebot came from IP addresses outside those companies' published ranges. Unverified claims for GoogleOther, ExaSearchBot and Baiduspider are dropped altogether, and other Cloudflare categories like SEO tools, webhooks, link previews, accessibility, advertising and security bots are left out.

Where crawler traffic shows up in Clicktag

Instead of mixing bots into your pageviews or throwing the data away completely, Clicktag puts crawler activity on the Crawlers tab in your dashboard.

To turn it on, your site must be verified in Clicktag and proxied through Cloudflare on any plan, including the Cloudflare Free plan. You do not add any extra code to your site:

1. Open your site's settings in Clicktag, go to Integrations and find Cloudflare.
2. Click Create a read-only token. Cloudflare opens with Zone Read and Analytics Read preselected.
3. Copy the token, paste it into Paste the token in Clicktag and click Connect.

Clicktag imports the last 30 days of crawler history on connect, then syncs every 15 minutes. While Cloudflare only keeps this analytics data for 31 days, Clicktag stores the hourly aggregates so your history keeps growing after Cloudflare drops it.

On the Crawlers tab, you get four headline numbers: AI crawler hits, Search engine hits, Pages crawled and Unverified hits. The Bots panel lists the top 100 bots with their verified hits, last seen hour, unverified claims, and a blocked count showing verified hits your site answered with HTTP 401, 403 or 429. That blocked count makes it easy to see if a firewall rule or bot setting is turning crawlers away.

Only page and text requests to your site's hostname and its www twin count. Scripts, styles, fonts, images and /cdn-cgi/ endpoints are ignored, while text files like /robots.txt and /llms.txt count as pages. Read the AI crawlers documentation for the full API and setup details.

FAQ

Does Clicktag block bad bots or AI crawlers?

No. Clicktag reports crawler activity and shows how many verified hits returned 401, 403 or 429 status codes, but blocking happens in your Cloudflare settings or robots.txt file.

Do AI crawlers trigger goals or custom events?

No. AI crawlers like GPTBot, ClaudeBot and PerplexityBot fetch raw HTML and do not run JavaScript, so they never load latest.js or fire window.tt_event. Search crawlers that do run JavaScript are detected as bots by their user agent, and bot hits are left out of every report.

What happens when a real person clicks a link inside ChatGPT or Perplexity?

That is a human visit, not bot traffic. When someone clicks a link in ChatGPT, their browser loads your page, runs latest.js, and shows up in your main Visitors and Sources reports (often with utm_source=chatgpt.com).

Does crawler tracking work without Cloudflare?

The main web analytics script works on any host, and hits it detects as bots are left out of your reports. The Crawlers tab specifically requires your domain to be proxied through Cloudflare, because Clicktag reads those raw HTML requests from Cloudflare's zone analytics using a read-only token.

How many sites and how much history are included for free?

Clicktag is free with no credit card required, and you can add up to 50 websites per account. Connecting Cloudflare imports 30 days of crawler history at 1-hour granularity and syncs every 15 minutes, keeping up to the top 100 bots and top 100 pages per report.