Clicktag ChatGPT search crawler guide
OAI-SearchBot: the crawler that puts you in ChatGPT search
Clicktag is a free web analytics tool that tracks verified OAI-SearchBot hits per page and human referral visits from chatgpt.com. Connect Cloudflare to import 30 days of crawler history with a 15-minute sync, no cookies and no credit card.
OAI-SearchBot is OpenAI's web crawler for search. It fetches and indexes web pages so your site can appear as a cited source in ChatGPT search answers. It is separate from OpenAI's training crawler, and you control it independently in your robots.txt file. Because OAI-SearchBot reads raw HTML and never executes JavaScript, standard browser analytics scripts never record its visits.
What OAI-SearchBot does
OpenAI uses a separate user agent for each job. OAI-SearchBot exists specifically to surface websites in ChatGPT search features. Its user agent string ends with compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot, and OpenAI publishes its IP ranges at openai.com/searchbot.json.
According to OpenAI's official bot documentation: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
If you opt out of OAI-SearchBot in
robots.txt, your site will not be shown in ChatGPT search answers, though it can still appear as a navigational link.
OAI-SearchBot vs. GPTBot vs. ChatGPT-User
The settings for GPTBot and OAI-SearchBot are independent, so you can opt out of training without leaving ChatGPT search. OpenAI also runs ChatGPT-User, which visits pages for user actions in ChatGPT and Custom GPTs.
| Bot name | Purpose | robots.txt behavior | What it means for your site | Published IP list |
|---|---|---|---|---|
| OAI-SearchBot | Surfaces websites in ChatGPT search features | Honors robots.txt; updates take ~24 hours |
Blocking it removes your site from ChatGPT search answers (navigational links may still appear) | openai.com/searchbot.json |
| GPTBot | Crawls content that may be used to train OpenAI's generative AI foundation models | Honors robots.txt; independent of OAI-SearchBot |
Disallowing it signals your content should not be used for model training | openai.com/gptbot.json |
| ChatGPT-User | Visits pages for user actions in ChatGPT and Custom GPTs | Not used for automatic crawling; robots.txt rules may not apply |
Not used to determine whether content appears in Search | openai.com/chatgpt-user.json |
| PerplexityBot | Surfaces and links websites in Perplexity search results | Honors robots.txt; updates may take up to 24 hours |
Not used to crawl content for AI foundation models | perplexity.ai/perplexitybot.json |
| Claude-SearchBot | Navigates the web to improve Claude search result quality | Honors robots.txt (set per subdomain) and Crawl-delay |
Blocking it prevents indexing for Claude search and may reduce visibility | claude.com/crawling/bots.json |
Allow OAI-SearchBot while blocking GPTBot in robots.txt
If you want your pages to show up in ChatGPT search results, but you do not want OpenAI using your pages to train foundation models, put separate rules in your robots.txt file:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
OpenAI states that it takes about 24 hours from a robots.txt update for its systems to adjust.
OAI-SearchBot does not run JavaScript
OAI-SearchBot fetches the raw HTML your server sends and never runs client-side JavaScript. A December 2024 study by Vercel and MERJ found that none of the major AI crawlers render JavaScript, including OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot and PerplexityBot.
This creates two practical problems for site owners:
- Your content must be in the server response. If your site relies on client-side rendering to populate text, headings or links in the browser, OAI-SearchBot sees an empty shell. Use server-side rendering or static HTML for any page that matters for ChatGPT SEO.
- Browser analytics scripts never fire. Because the crawler never runs
<script>tags, JavaScript trackers cannot see OAI-SearchBot at all.
Check if your firewall is blocking OAI-SearchBot
Having Allow: / in robots.txt is not enough if a CDN firewall rule, rate limit or bot protection setting answers OpenAI's requests with HTTP 401, 403 or 429.
Clicktag's AI crawler analytics reads crawler requests directly from Cloudflare using a read-only token (Zone Read and Analytics Read), with no code needed on your site. It works on any Cloudflare plan, including Free.
- Open your site's settings in Clicktag, go to Integrations and find Cloudflare.
- Click Create a read-only token, create the token in Cloudflare with Zone Read and Analytics Read, and copy it.
- Paste it into Paste the token and click Connect.
Clicktag imports your last 30 days of crawler history on connect, then syncs every 15 minutes. Cloudflare only keeps this data for 31 days, while Clicktag stores it per hour so your history keeps growing.
On the Crawlers tab, Clicktag separates verified hits from unverified ones. Anyone can put OAI-SearchBot in a user agent string, so Clicktag checks whether Cloudflare verified the bot or whether the request IP belongs to OpenAI's published ranges (the IP check runs during the sync and IP addresses are never stored).
The Bots panel shows verified hits, when each bot was last seen, and a blocked count for verified hits your site answered with 401, 403 or 429. Pages lists the top 100 paths with how many different bots read each one; click OAI-SearchBot to filter the tab and confirm it reaches your key URLs. See the AI crawlers documentation for details.
Indexing hits vs. ChatGPT referral traffic
An OAI-SearchBot hit is not a visitor. It is OpenAI's server fetching your HTML so it can index or cite your page later.
When a real person reads an answer in ChatGPT and clicks a link to your site, their browser loads your page and runs Clicktag's latest.js script (about 4 KB). ChatGPT adds utm_source=chatgpt.com to links it shows. Because Clicktag reads utm_source first when attributing traffic sources, those visits appear under the chatgpt.com source, separate from crawler activity.
A site can have up to 50 goals and funnels combined, where a goal counts visitors who view a page or fire a custom event. Break a goal down by source to compare chatgpt.com visitors with google.com visitors. Goals live in your own dashboard; the live demo shows the source reports with sample data.
FAQ
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI keeps GPTBot and OAI-SearchBot rules independent. Disallowing GPTBot signals that your content should not be used to train OpenAI's foundation models, while allowing OAI-SearchBot keeps your pages eligible for ChatGPT search answers.
How long does OAI-SearchBot take to pick up robots.txt changes?
OpenAI states that it takes about 24 hours after you update your robots.txt file for its systems to adjust.
Can Clicktag block OAI-SearchBot or other crawlers?
No. Clicktag reads Cloudflare's analytics to show you verified hits, status codes and blocked counts per bot, but it does not block bots or modify your traffic. You manage blocking through robots.txt or your Cloudflare firewall settings.
Do I need a paid Cloudflare plan or a credit card to use Clicktag?
No. Clicktag is free with no credit card required (you sign in with a Google account and can track up to 50 websites), and the Cloudflare crawler integration works on the Cloudflare Free plan as long as your domain is proxied through Cloudflare.
What are the limits of Clicktag's crawler tracking and web analytics?
Crawler tracking requires your domain to be verified in Clicktag and proxied through Cloudflare, counts only your hostname and its www twin, imports 30 days of history on connect, syncs every 15 minutes at 1-hour granularity, and shows the top 100 bots and top 100 pages per report using Cloudflare's sample-adjusted estimates. For visitor analytics, Clicktag sets no cookies, stores no IP addresses, and does not offer session recordings, heatmaps, bounce rate or revenue tracking.
Updated October 30, 2026
