How to block AI crawlers, and check that they listened

To block AI crawlers from collecting your site for model training, add Disallow: / rules for training bots like GPTBot, ClaudeBot, and the Google-Extended token in your robots.txt file, or block them at the edge with Cloudflare's AI bot policies. Leave search bots like OAI-SearchBot and PerplexityBot allowed if you still want your pages cited in AI answers. After about 24 hours, open the Crawlers tab in Clicktag to confirm verified hits dropped or now show up as blocked.

Decide what to block: training, search, or user fetchers

AI companies run separate bots for different jobs. Blocking every AI user agent at once can cut off referral traffic from AI answer engines along with the training crawls.

• Training crawlers collect pages that can be used to train AI models. This group includes GPTBot, ClaudeBot, CCBot, Amazonbot, MistralAI-Training, and the robots.txt tokens Google-Extended and Applebot-Extended. Disallowing them signals that your pages should stay out of their training data.
• Search crawlers index pages to show links and citations in AI search products. Blocking OAI-SearchBot removes your site from ChatGPT search answers (though navigational links can still appear). PerplexityBot surfaces and links sites in Perplexity search results, and Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude's search results.
• User-triggered fetchers load a page because a person asked the assistant for it, not as part of an automatic crawl. ChatGPT-User, Perplexity-User, Claude-User, and Amzn-User fall here.

A robots.txt template that blocks training and keeps search

Place this in /robots.txt at the root of each subdomain you want to protect. Anthropic asks for the opt-out on every subdomain.

# Block AI model training crawlers and tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: meta-externalagent
Disallow: /

# Allow AI search crawlers so pages still appear in AI answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
Allow: /

Google-Extended and Applebot-Extended are robots.txt tokens. Google still crawls with Googlebot and Apple with Applebot, so you won't see Google-Extended in your request logs.

Why user-triggered fetchers still show up

Even with a strict robots.txt, you can still see hits from ChatGPT-User, Perplexity-User, and Amzn-User, because a person asked for the page.

OpenAI states that for ChatGPT-User, "robots.txt rules may not apply." Perplexity's documentation notes that Perplexity-User "generally ignores robots.txt rules." Amazon says Amzn-User "may not follow all robots.txt directives." If you want to stop user-triggered fetchers too, you need a firewall rule rather than robots.txt.

Blocking AI crawlers at the edge with Cloudflare

robots.txt is voluntary. Well-behaved bots follow it, while scrapers and spoofed user agents can ignore it. If your domain is proxied through Cloudflare, you can block requests at the edge before they reach your server.

• AI Crawl Control (formerly AI Audit): available on all Cloudflare plans. It monitors AI crawler activity, sets allow or block rules per crawler, and tracks whether crawlers follow your robots.txt. Pay per crawl is in private beta.
• Configure AI bot policies: Cloudflare deprecated its one-click "Block AI bots" setting on September 15, 2026. Its replacement under Security Settings > Configure AI bot policies has Search, Agent, and Training presets, each set to Block (on all pages), Block on pages with ads, or Allow. From September 15, 2026, new domains default to blocking Training and Agent bots on pages that display ads, with Search allowed. Crawlers that mix search and training are blocked by every configuration that blocks AI training.
• Managed robots.txt: with Bot Preference Sync on, Cloudflare prepends managed directives and its Content Signals Policy to your /robots.txt when at least one Search, Agent, or Training policy is set to Block, Block on pages with ads, or (for Training) Disallow.

robots.txt only works for bots that choose to follow it, while a Cloudflare block refuses the request before it reaches your server.

Check that your block worked in Clicktag

JavaScript analytics scripts never see AI crawlers, because GPTBot, ClaudeBot, and PerplexityBot fetch raw HTML and never run JavaScript, so latest.js never fires for them. To see whether bots respected your changes, use Clicktag's AI crawler analytics.

Connect a read-only Cloudflare token (Zone Read and Analytics Read) under Integrations > Cloudflare in your Clicktag site settings. Clicktag imports your last 30 days of crawler history and syncs every 15 minutes, on any Cloudflare plan including Free.

Wait at least a day before judging a robots.txt change. OpenAI says OAI-SearchBot takes about 24 hours to pick it up, Perplexity says up to 24 hours for PerplexityBot, Amazon says about 24 hours, and DuckDuckGo says a DuckAssistBot opt-out takes effect after 72 hours. Then open the Crawlers tab in Clicktag:

1. Check the Bots panel for each crawler you disallowed: its verified hits and when it was last seen, the latest hour with a verified hit.
2. Check each bot's blocked count: verified hits your site answered with HTTP 401, 403, or 429, so a firewall rule or Cloudflare bot setting that turns crawlers away is easy to spot.
3. Click /robots.txt in the Pages list to filter the tab to that file and see which bots fetched it.
4. Look at the unverified count. A hit is verified when Cloudflare verified the bot, or when it came from an IP range the bot's company publishes; Clicktag checks the lists of OpenAI, Perplexity, Google, Microsoft, Apple, Common Crawl, and DuckDuckGo. A request that claims to be GPTBot without being verified is kept apart, so a scraper using GPTBot's name won't make you think OpenAI ignored your robots.txt.

FAQ

Does Clicktag block AI crawlers itself?

No. Clicktag reads your Cloudflare analytics with a read-only token and shows crawler hits per bot and per page, with blocked and unverified counts. It never changes your site or your Cloudflare settings, and it never blocks traffic.

How long does it take for AI bots to obey a new robots.txt rule?

OpenAI says about 24 hours for OAI-SearchBot, Perplexity says up to 24 hours for PerplexityBot, and Amazon says about 24 hours. DuckDuckGo says a Disallow for DuckAssistBot takes effect after 72 hours.

Should I block AI crawlers by IP address instead?

Anthropic says blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because it stops its bots from reading your robots.txt. Use robots.txt for bots that follow it, and Cloudflare's AI bot policies or AI Crawl Control to block at the edge.

Does blocking Google-Extended hurt my Google Search rankings?

No. Google says Google-Extended controls whether content it crawls may be used to train Gemini models and for grounding, and that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

What are the limits on Clicktag's AI crawler tracking?

Your site must be verified in Clicktag and proxied through Cloudflare (any plan, including Free). Only requests to your site's hostname and its www twin are counted, and reports show the top 100 bots and top 100 pages at 1-hour granularity. Connecting imports 30 days of history, and disconnecting your token or clicking Stop tracking deletes that crawler history.