See when Googlebot crawls your site, page by page, free

Clicktag's visitor reports never show Googlebot, because its script recognizes search engine crawlers by their user agent and leaves them out of visits and pageviews. To see when Googlebot crawls your site without digging through raw server logs, connect Cloudflare to Clicktag. The Crawlers tab imports 30 days of request history, checks every Googlebot hit against Cloudflare's bot verification and Google's published IP ranges, and breaks down verified crawls per page with a sync every 15 minutes.

What Googlebot is and why JavaScript analytics miss it

Googlebot is Google's standard web crawler for Google Search (including Discover and all Search features), Google Images, Video mode and Google News. Google's common crawlers always obey robots.txt rules for automatic crawls.

Even when Googlebot executes JavaScript on a page, Clicktag recognizes its user agent and drops the hit so bot traffic does not inflate your human visitor counts. Bots are excluded from visits and pageviews by design, and AI bots like GPTBot or ClaudeBot never run JavaScript at all.

Because the browser script filters bots out, you have to read requests at the edge before the page loads. If your domain is proxied through Cloudflare (on any plan, including Free), Cloudflare sees every request it proxies. Clicktag reads the requests to your hostname and its www twin with a read-only Cloudflare API token and displays them on the Crawlers tab, with no extra code on your site.

Fake Googlebot user agents vs verified hits

Anyone can send an HTTP request with Googlebot in the user agent string. On Clicktag's own sites, unverified hits claiming to be Googlebot came from IPs outside Google's published ranges. If you only filter server logs by the user agent string, your numbers will include fake hits.

A request that claims to be Googlebot without passing Cloudflare verification or matching Google's published IP list is counted separately as unverified.

Clicktag separates real crawls from spoofed requests during every 15-minute sync:

• Verified Googlebot hits: Cloudflare verified the request in its Search Engine Crawler category, or the request came from Google's published common crawler IP ranges. Clicktag groups Googlebot, Googlebot-Image, Googlebot-Video and Googlebot-News under Googlebot.
• Unverified hits: The user agent claimed to be Googlebot, but the request failed Cloudflare verification and came from an IP address outside Google's published ranges.
• IP privacy: Clicktag checks IP addresses against published lists during the sync and never stores IP addresses.
• GoogleOther and Google-Extended: Clicktag tracks GoogleOther (Google's generic crawler for its product teams, for example for internal research crawls) as a separate AI training bot, and ignores unverified GoogleOther claims. Google-Extended has no separate user agent string; it is only a robots.txt token.

How to check Googlebot crawls on your site

You can set up crawler tracking in the dashboard if your site is verified in Clicktag and proxied through Cloudflare:

1. Open your site settings in Clicktag, go to Integrations and find Cloudflare.
2. Click Create a read-only token. Cloudflare opens with Zone Read and Analytics Read pre-selected for your zones.
3. Create the token in Cloudflare, copy it, paste it into Paste the token in Clicktag and click Connect.

On connect, Clicktag imports the last 30 days of crawler history, one week at a time. Once the import finishes, it syncs every 15 minutes and re-reads the previous hour to catch late data. Because Cloudflare only keeps request analytics for 31 days on Free zones, Clicktag stores your hourly crawler counts so your history keeps growing after Cloudflare drops it.

On the Crawlers tab, search engine crawlers start hidden on the main chart; click search engines in the chart legend to show them. In the Bots panel, click Googlebot to filter the entire tab:

• Verified hits and last seen: total verified requests and the latest hour with a verified Googlebot hit.
• Blocked count: verified Googlebot requests your server answered with HTTP 401, 403 or 429, so you can spot a firewall rule turning Google away.
• Pages: the top 100 paths crawled by Googlebot. Static assets (scripts, styles, fonts, images) and /cdn-cgi/ endpoints are excluded, while text files like /robots.txt and /llms.txt count as pages.
• Responses: the HTTP status codes your site returned to verified crawlers.

What you see where in Clicktag

Clicktag keeps bot requests, human visits, and search performance in separate views so none of them pollute the others. The live demo shows the visitor reports with sample data.

Pair Googlebot crawls with Search Console clicks and impressions

Seeing that Googlebot crawled a new page is half the job; the next question is whether Google is showing that page in search results and sending visitors.

In site settings under Integrations, click Connect Google Search Console to link your verified property. Clicktag backfills up to 16 months of Web search history and refreshes the trailing 10 days every 6 hours. Like crawler history, Clicktag stores your synced Search Console days so your records outlive Google's 16-month retention window.

With both connected, you can compare crawler and Search Console data side by side (see the crawler docs and Search Console docs):

• Check the Crawlers tab to confirm Googlebot fetched your new path and got a 200 status code rather than a 403 or 404.
• Open the Search Console view and filter by that path (f=page:/your-path) to see its impressions, clicks, average position, and the queries that showed it.
• Compare Google's reported search clicks with the google.com pageviews Clicktag's tracker measured on the same day.

FAQ

Does Clicktag block Googlebot or fake Googlebot requests?

No. Clicktag reads request analytics from Cloudflare using a read-only token and never changes your site or your Cloudflare firewall settings. It shows you the verified, unverified and blocked (401, 403, 429) counts so you can adjust rules in Cloudflare or robots.txt yourself.

Do I need a paid Cloudflare plan to see Googlebot crawls?

No. The Cloudflare analytics endpoints that Clicktag reads are available on all Cloudflare plans, including the Free plan, as long as your domain's traffic is proxied through Cloudflare.

How does Clicktag verify that a Googlebot hit is real?

A hit counts as verified when Cloudflare classifies it as a verified Search Engine Crawler or when the request IP matches Google's published common crawler IP list. Requests with a Googlebot user agent that fail both checks are counted under unverified hits.

Why don't CSS, JavaScript or image requests show up in the Googlebot count?

Clicktag filters out scripts, stylesheets, fonts, images and Cloudflare's /cdn-cgi/ endpoints so the numbers reflect actual page and document fetches. HTML pages and text files such as /robots.txt and /llms.txt are counted, on your site's hostname and its www twin.

Can I query Googlebot crawl counts through an API?

Yes. You can call GET /api/sites/{hostname}/crawlers/overview with a read-scoped management key (tt_mk_) and filter specifically for Googlebot using f=bot:Googlebot or for all search engines using f=purpose:search.