GPTBot: what OpenAI crawls and how to see its visits

GPTBot is OpenAI's web crawler for collecting public web pages that may be used to train generative AI foundation models. It is separate from OAI-SearchBot, which surfaces sites in ChatGPT search answers, and ChatGPT-User, which visits pages for user actions in ChatGPT and Custom GPTs. You can block GPTBot in your robots.txt file without losing visibility in ChatGPT search, and you can track its verified visits through Cloudflare in Clicktag.

What is GPTBot and what is its user agent?

OpenAI uses separate user agent tokens, each with its own IP list, for training, search and user actions. GPTBot is the training crawler. When it requests a page, it sends this user agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

OpenAI documents these tokens on its official bots page and publishes the IP ranges for each as a JSON file:

Should you block GPTBot in robots.txt?

Whether you block AI crawlers depends on whether you want your pages in future training datasets. Per OpenAI's documentation, disallowing GPTBot signals that your site's content should not be used to train generative AI foundation models.

Blocking GPTBot in robots.txt does not remove your pages from ChatGPT search results, because GPTBot and OAI-SearchBot settings are independent.

If you opt out of OAI-SearchBot, your site will not be shown in ChatGPT search answers, though it can still appear as navigational links. If you only disallow GPTBot and leave OAI-SearchBot allowed, your pages stay eligible for ChatGPT search while opting out of training crawls. OpenAI notes that it can take about 24 hours after a robots.txt update for ChatGPT search to adjust.

To block GPTBot across your entire site, add this rule to your robots.txt file:

User-agent: GPTBot
Disallow: /

To block GPTBot from training on your site while explicitly allowing OpenAI to surface your pages in ChatGPT search, set rules for both tokens:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Why Google Analytics and JavaScript trackers never show GPTBot

If you open Google Analytics or any script-based tracker to check how often OpenAI crawls your site, you'll see zero hits from GPTBot. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot fetch the raw HTML of a page and never run JavaScript.

Because the browser script never runs, client-side analytics tools never fire a hit. Search engine crawlers that do run JavaScript are recognized as bots by their user agent, and Clicktag leaves them out of your visits. To see crawler requests, you have to read them at the network edge or in your server logs.

How to see verified GPTBot visits in Clicktag

Clicktag includes free AI crawler analytics on the Crawlers tab of your dashboard. Instead of relying on JavaScript, Clicktag reads request data from Cloudflare's analytics using a read-only Cloudflare token. It works on any Cloudflare plan, including the Free plan, and requires no new code on your site.

1. Verify your site in Clicktag

Sign in at clicktag.app with a Google account and verify your site using the 4 KB latest.js script tag, a DNS TXT record, or a file under /.well-known/. Your domain must be proxied through Cloudflare so Cloudflare sees the requests.

2. Create a read-only Cloudflare token

Open your site's settings in Clicktag, go to Integrations, and find Cloudflare. Click Create a read-only token. Cloudflare opens its token creation page with Zone Read and Analytics Read already selected for your zones.

3. Connect and import 30 days of history

Copy the token, paste it into Paste the token in Clicktag, and click Connect. One token covers all verified sites in your account that match its zones.

Once connected, Clicktag imports the last 30 days of crawler history and then syncs every 15 minutes. While Cloudflare drops this analytics data after 31 days, Clicktag stores the hourly counts so your history keeps growing over time. You can read the full technical setup in the AI crawlers documentation.

Why fake GPTBot user agents show as unverified

Anyone running a scraper can put GPTBot/1.4 into their HTTP user agent header. If you only filter server logs by user agent string, scrapers pretending to be OpenAI will pollute your counts.

Clicktag separates verified hits from unverified ones. A GPTBot hit only counts as verified when Cloudflare verified the bot in its AI Crawler, AI Search, AI Assistant or Search Engine Crawler category, or when the request came from an IP range OpenAI publishes. Clicktag checks the IP addresses during the 15-minute sync and never stores IPs.

If a request claims to be GPTBot in its user agent string but comes from an IP address outside OpenAI's published ranges and fails Cloudflare verification, Clicktag counts it as unverified. On the Crawlers tab, you'll see:

• Verified hits: listed per bot in the Bots panel, with the latest hour a verified hit came in.
• Blocked count: verified hits where your site returned a 401, 403 or 429 status code, so you can tell right away if a firewall rule or Cloudflare bot policy is turning GPTBot away.
• Unverified count: requests that claimed the GPTBot name without matching OpenAI's IPs or Cloudflare verification.
• Pages: the top 100 paths crawled on your hostname and its www twin, with how many different bots read each one, including text files like /robots.txt and /llms.txt (static assets like scripts, styles, fonts and images are excluded).

FAQ

Does Clicktag block GPTBot for me?

No. Clicktag reports crawler activity, status codes and blocked request counts, but it does not block or allow bots. To block GPTBot, update your robots.txt file or configure AI bot policies in your Cloudflare security settings.

Do I need a paid Cloudflare or Clicktag plan to track GPTBot?

No. Clicktag is free with no credit card required and supports up to 50 websites per account. The Cloudflare integration works on all Cloudflare plans, including the Cloudflare Free plan, as long as your domain is proxied through Cloudflare.

If I block GPTBot, will my site still get traffic from ChatGPT links?

It can. Blocking GPTBot only opts out of training crawls; it does not stop OAI-SearchBot from surfacing your site in ChatGPT search or ChatGPT-User from fetching pages for users. ChatGPT adds utm_source=chatgpt.com to the links it shows, so Clicktag lists those human visits under the chatgpt.com source.

Which other bots does Clicktag track alongside GPTBot?

Along with OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Clicktag tracks ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, GoogleOther, Bingbot, Applebot, Amazonbot, CCBot, DuckDuckBot, DuckAssistBot, meta-externalagent, meta-externalfetcher, Bytespider, MistralAI-User, ExaSearchBot and Baiduspider.

What happens to my crawler history if I disconnect Cloudflare?

Clicking Stop tracking in site settings deletes the crawler history for that site, and clicking Disconnect removes your Cloudflare token and deletes crawler history across all your sites. Both ask for confirmation first, and neither touches your human web analytics data.