Applebot and Applebot-Extended: what Apple crawls and why

Applebot is Apple's web crawler. Its data powers search in Spotlight, Siri and Safari, and may also be used to train Apple's foundation models and to give AI-generated answers context. Applebot-Extended is not a separate crawler; it is a robots.txt token you disallow to opt your content out of Apple's foundation model training, while your pages can still appear in Siri, Spotlight and Safari search results.

Applebot vs Applebot-Extended

Apple uses one crawler to fetch pages and a second token in robots.txt to control how that fetched data is used. This works the same way as Googlebot and Google-Extended.

Read Apple's official Applebot documentation for the full specification.

How Applebot reads your robots.txt rules

Applebot has a fallback behavior that catches many site owners off guard. If your robots.txt file does not mention Applebot at all, but it does have a section for Googlebot, Applebot follows the Googlebot instructions.

Applebot also does not follow the Crawl-delay directive, so a crawl delay under User-agent: * or User-agent: Applebot has no effect. Apple says Applebot adjusts its crawl rate automatically when a site slows down or returns errors.

If you want your pages to show up in Safari, Siri and Spotlight search, but you do not want Apple using your text to train its generative foundation models, allow Applebot and disallow Applebot-Extended:

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

Disallowing Applebot-Extended opts your content out of training Apple's foundation models, and those pages can still be included in Siri, Spotlight and Safari search results.

What if you want search links in Siri and Spotlight, yet you want to keep your pages out of the AI-generated answers to broad world knowledge questions in Siri and Search? According to Apple's documentation, you leave Applebot allowed and add the standard nosnippet meta tag to those pages.

If you want to stop Apple from crawling your site for any reason at all, disallow Applebot:

User-agent: Applebot
Disallow: /

For rules covering OpenAI, Anthropic and Perplexity alongside Apple, see our guide on how to block AI crawlers and the full AI crawler list.

How to verify real Applebot traffic

Anyone can send an HTTP request with Applebot in the User-Agent header. Apple documents two ways to verify that a request actually came from Applebot:

• Reverse DNS: the IP address resolves to a host in the *.applebot.apple.com domain, and that host resolves back to the same IP.
• Published IP ranges: the IP matches a CIDR prefix in search.developer.apple.com/applebot.json.

In Clicktag, a request counts as a verified Applebot hit if Cloudflare verified it in its AI Crawler, AI Search, AI Assistant or Search Engine Crawler category, or if it came from an IP range on Apple's published list. Any other request claiming to be Applebot is counted as unverified.

Why your visitor reports don't show Applebot

Browser analytics, Clicktag's latest.js script included, count what runs in a visitor's browser. Apple says Applebot may render your pages in a browser, but Clicktag recognizes bot user agents such as Applebot's and leaves them out of visits and pageviews.

Because of that, your regular traffic charts won't show how often Applebot visits, which pages it reads, or whether your server is returning 403 or 429 status codes to it. You need server-side or edge request logs to see crawler traffic.

Track Applebot per page for free with Clicktag

Clicktag includes AI crawler analytics at no cost. Instead of asking you to ship server logs or install middleware, Clicktag reads crawler requests from Cloudflare's analytics using a read-only API token. It works on any Cloudflare plan, including the Free plan.

Here is the setup from the crawler docs:

1. Verify your site in Clicktag (with the latest.js tag on your homepage, a DNS TXT record, or a file under /.well-known/) and make sure its domain is proxied through Cloudflare.
2. Open your site settings in Clicktag, go to Integrations, and find Cloudflare.
3. Click Create a read-only token. Cloudflare opens with Zone Read and Analytics Read pre-selected. Create and copy the token.
4. Paste the token into Paste the token and click Connect.

On connect, Clicktag imports the last 30 days of crawler history, one week at a time, and then syncs new data every 15 minutes. While Cloudflare drops this analytics data after 31 days, Clicktag stores the hourly aggregates so your crawler history keeps growing over time.

On the Crawlers tab, you get:

• Bots panel: Verified hits for Applebot, GPTBot, ClaudeBot, Googlebot and others, plus the latest hour each bot was seen.
• Blocked and unverified counts: See how many verified Applebot requests your site answered with 401, 403 or 429, and how many requests claimed to be Applebot without passing verification.
• Pages: The top 100 paths crawled, with how many different bots read each, including /robots.txt and /llms.txt (static assets like scripts, styles, fonts and images are excluded).
• Responses: The HTTP status codes verified crawlers received.

Clicktag checks IP addresses against Apple's published list during the 15-minute sync and never stores IP addresses. It is a reporting tool; it shows you what Applebot is doing on your site, but it does not block bots.

FAQ

Will Applebot-Extended show up in my crawler logs?

No. Apple says Applebot-Extended does not crawl webpages; it is only a robots.txt token that decides how data crawled by Applebot may be used. Those crawls show up as Applebot.

Why is Applebot crawling paths I disallowed for User-agent: *?

If your robots.txt file has a section for Googlebot that allows those paths, and you do not have a dedicated User-agent: Applebot section, Applebot follows the Googlebot rules instead of the wildcard rules. Add an explicit User-agent: Applebot block to give it separate instructions.

Does Clicktag block Applebot or manage robots.txt?

No. Clicktag only reports crawler activity; it does not modify your robots.txt file or block traffic. You can block or allow crawlers in your own robots.txt file or in Cloudflare's AI Crawl Control settings, then use Clicktag's blocked counter (HTTP 401, 403 and 429 responses) to confirm your rules are working.

What are the limits on Clicktag's crawler tracking?

Clicktag is free with no credit card required and supports up to 50 websites per account. For crawler reports, your domain must be proxied through Cloudflare (any plan, including Free); Clicktag imports 30 days of history on connect, syncs every 15 minutes at 1-hour granularity, and shows the top 100 bots and top 100 pages per report using Cloudflare's sample-adjusted estimates.

Does Clicktag track visitors from Apple devices too?

Yes. When real people visit your site from Safari on iOS or macOS, Clicktag's script (about 4 KB) records their pageviews, referrers, operating systems and browsers without cookies or storing IP addresses, while keeping bot hits out of your visitor counts. The live demo shows these reports with sample data.