Google-Extended: the robots.txt token for Gemini, explained

Google-Extended is a robots.txt product token, not a crawler. It has no separate user agent string and never makes HTTP requests of its own, so you will never see Google-Extended in your server logs or on your analytics dashboard. Instead, it tells Google whether content fetched by its existing crawlers, such as Googlebot, may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI.

Why Google-Extended never shows up in your logs

Several AI companies run separate bots for search and AI training, like OpenAI's GPTBot and OAI-SearchBot. Each one sends its own user agent string in the request headers, which makes them show up as distinct entries in your logs.

Google handles this differently. Googlebot fetches the page, and Google-Extended is only a rule inside your /robots.txt file that sets permissions for how that crawled data gets used downstream. Because Google never sends Google-Extended as a user agent, it will not appear in Cloudflare, in your raw access logs, or on the Crawlers tab in Clicktag.

Google-Extended is a permission rule in robots.txt, while Googlebot is the crawler that actually fetches your pages.

What Google-Extended controls (and what it leaves alone)

According to Google's documentation on common crawlers, adding a Google-Extended rule controls two things:

• Whether content Google crawls can be used to train future Gemini models that power Gemini Apps and the Vertex AI API for Gemini.
• Whether content can be used for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.

It does not block your pages from Google Search. Google states directly that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal".

AI Overviews in Google Search are part of Search, not Gemini Apps. Because Googlebot crawls for Google Search and all Search features, disallowing Google-Extended does not opt your site out of AI Overviews. And when a human clicks a link to your site from an AI Overview, Clicktag groups Google search domains under google.com, while referrals from gemini.google.com show up as a separate traffic source.

Googlebot vs. Google-Extended vs. GoogleOther

Google runs more than one token in robots.txt. Here is how the three names compare:

How to block Google AI training in robots.txt

If you want Google to keep indexing your site for Search while opting out of Gemini model training and grounding, leave Googlebot allowed and disallow Google-Extended in your /robots.txt file:

User-agent: Google-Extended
Disallow: /

If you also want to block other AI crawlers like OpenAI's GPTBot, Anthropic's ClaudeBot and Common Crawl's CCBot, list each one under its own User-agent line or see our guide to block AI crawlers:

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

If your domain uses Cloudflare, you do not have to edit the file by hand. Cloudflare's managed robots.txt is available on all plans, including Free. When you turn on Bot Preference Sync and set a Search, Agent or Training policy to block, Cloudflare prepends managed directives and a Content Signals Policy (defaulting to search=yes, ai-train=no) to your /robots.txt automatically.

Seeing verified Googlebot and GoogleOther hits in Clicktag

Even though Google-Extended makes no requests, you can check two things: whether bots are reading your /robots.txt file, and what Googlebot and GoogleOther are fetching across your site.

Clicktag's script never counts crawlers as visits: AI crawlers fetch raw HTML without running JavaScript, and search engine crawlers that do run it are recognized by their user agent and left out. Clicktag includes AI crawler analytics that reads crawler requests from Cloudflare's analytics using a read-only token, with no code needed on your site.

To set it up on a verified Clicktag site whose domain is proxied through Cloudflare:

1. Open your site's settings, go to Integrations and find Cloudflare.
2. Click Create a read-only token. Cloudflare opens its token page with Zone Read and Analytics Read already selected. Create the token and copy it.
3. Paste it into Paste the token and click Connect.

Clicktag imports the last 30 days of crawler history on connect, then syncs every 15 minutes. Cloudflare drops this data after 31 days, but Clicktag keeps your hourly history growing after Cloudflare has dropped it.

On the Crawlers tab, you get:

• Verified vs. unverified hits: Anyone can put Googlebot in a user agent header. Clicktag checks requests against Cloudflare's bot verification and Google's published IP ranges during the sync (IP addresses are never stored). Unverified claims of Googlebot are kept separate, and unverified claims of GoogleOther are left out.
• Pages: The top 100 paths, with how many different bots read each one. While scripts, styles, fonts and images don't count, text files like /robots.txt and /llms.txt do count as pages, so you can see whether Googlebot or GoogleOther read your /robots.txt.
• Blocked counts and responses: See how many verified requests your site answered with 401, 403 or 429 status codes.

Clicktag shows crawler activity so you can audit what bots do, though it does not block bots itself. See the full AI crawler list for the other bots it recognizes.

FAQ

Does Google-Extended have a user agent or IP list?

No. Google-Extended is only a product token you put in robots.txt. Google crawls with its existing user agents, such as Googlebot, and your Google-Extended rule controls how that content may be used.

Does blocking Google-Extended remove my site from Google AI Overviews?

No. Google AI Overviews are a Google Search feature, which is crawled by Googlebot. Google-Extended only controls Gemini model training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI, and it has no effect on Google Search inclusion or ranking.

Why don't I see Google-Extended on the Crawlers tab in Clicktag?

Because no crawler ever sends Google-Extended in an HTTP request header. You will see verified hits from Googlebot (under Search engine) and GoogleOther (under AI training), and you can click /robots.txt in Pages to filter the tab and see which Google bots read your rules.

Does Clicktag block GoogleOther or other AI crawlers?

No, Clicktag is an analytics tool and does not block traffic. It reads crawler requests from Cloudflare's analytics with a read-only token (Zone Read and Analytics Read) and shows you verified hits, unverified claims, and how many verified hits your site answered with 401, 403 or 429.

What are the limits of Clicktag's crawler tracking?

Your domain must be verified in Clicktag and proxied through Cloudflare (any plan, including Free, works). Clicktag imports 30 days of history on connect, syncs every 15 minutes at 1-hour granularity, and shows the top 100 bots and top 100 pages per report across up to 50 websites per account.