AI SEO: how to get your site cited by ChatGPT and Perplexity

To get your site cited by ChatGPT, Perplexity, Claude and Google AI Overviews, let their search crawlers read your pages, serve your text in the initial server HTML instead of waiting on client-side JavaScript, and put a direct answer with concrete facts at the top of each page. Google's documentation states that appearing in AI Overviews or AI Mode takes no special optimizations beyond standard Google Search indexing with a snippet.

What AI SEO is and what else people call it

AI SEO is the practice of making your website easy for AI answer engines to crawl, index and cite when someone asks a question. You'll see the same work under a few names:

• AI search optimization: optimizing for search inside ChatGPT, Perplexity and similar products.
• Generative engine optimization (GEO): formatting pages so generative models can quote accurate facts.
• Answer engine optimization (AEO): structuring pages around questions so answer engines can extract a clean answer.
• LLM SEO: making sure language models and their fetchers can read your pages.

What you control: let AI search crawlers in, put your text in the server HTML, and answer one question clearly per page.

Where each AI engine finds pages to cite

Most AI companies split their bots by job: one crawls for model training, another indexes pages for search results, and a third fetches a page live when a user asks a question.

A practical checklist for AI search optimization

1. Allow the AI search crawlers in robots.txt

If you block every AI bot in robots.txt to stop model training, you also shut yourself out of AI search answers. OpenAI documents that sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers (though they can still appear as navigational links), and that GPTBot and OAI-SearchBot settings are independent. OpenAI's systems take about 24 hours to adjust to a robots.txt update, and Perplexity says up to 24 hours for PerplexityBot.

Google-Extended controls whether Google may use your content for training Gemini models and for grounding in Gemini Apps and Vertex AI, and Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." To block training while staying in search answers, disallow the training bots and leave the search bots allowed:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Also check your firewall: a bot allowed in robots.txt can still get a 401, 403 or 429 at your CDN edge.

2. Put your content in the server HTML

A December 17, 2024 study by Vercel and MERJ found that none of OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider or PerplexityBot rendered JavaScript. In the same study, Gemini (which uses Googlebot's infrastructure) and Applebot did.

If your site is a single-page app that sends an empty <div id="root"></div> and fetches the text in the browser, those crawlers only see an empty shell. Render pages on the server or at build time so the full text, headings and tables are in the raw HTML.

3. Build one clear page per question

Give each question its own URL instead of burying 10 topics in one guide. Phrase the title and H1 like the question people ask, and answer it in the opening paragraph with the numbers, limits or prices the reader came for.

4. Use tables, plain headings and an FAQ

Put specs, limits, prices and bot names in standard HTML tables, use descriptive ## headings, and close the page with an FAQ that covers exact limits and edge cases. Keep facts and dates current, since live fetchers like ChatGPT-User read the page when someone asks about it.

5. Don't expect llms.txt to replace HTML and robots.txt

/llms.txt is a proposal published by Jeremy Howard on September 3, 2024 at llmstxt.org for a Markdown index file at your site root. OpenAI, Anthropic and Gemini publish llms.txt files for their own developer documentation, but none of the official crawler docs from Google, OpenAI or Anthropic say their crawlers or answer engines read it. All three document robots.txt as the control, so put your effort into clean server HTML first.

How to measure AI SEO for free with Clicktag

You also need to see whether AI bots read your site and whether people click the citations. Clicktag covers both for free, with no credit card and up to 50 websites per account.

1. Check AI crawler hits per page

Browser analytics scripts never see AI crawlers like GPTBot and ClaudeBot, because they fetch raw HTML and never run JavaScript. Clicktag's AI crawler analytics reads crawler requests from Cloudflare's analytics with a read-only token (Zone Read and Analytics Read), with no code on your site, on any Cloudflare plan including Free.

Connect Cloudflare under Integrations in your site's settings, and Clicktag imports the last 30 days of crawler history, then syncs every 15 minutes and keeps your hourly history after Cloudflare's 31-day window drops it. On the Crawlers tab, you get:

• Bots: verified hits for each bot, when it was last seen, and a blocked count of verified hits your site answered with 401, 403 or 429.
• Verified vs unverified kept apart: anyone can spoof a user agent, so a hit only counts as verified when Cloudflare verified the bot or the request came from an IP range published by OpenAI, Perplexity, Google, Microsoft, Apple, Common Crawl or DuckDuckGo.
• Pages: the top 100 paths crawled and how many different bots read each one, including /robots.txt and /llms.txt.

2. Track AI referral visits and landing pages

To see visitors who click through from an AI citation, install Clicktag's latest.js script (about 4 KB, no cookies, no IP storage). In your AI traffic analytics reports, ChatGPT, Perplexity, Claude, Gemini and Copilot appear as their own sources.

ChatGPT adds utm_source=chatgpt.com to links it shows, and Clicktag reads utm_source before the referring host, so those clicks show up under chatgpt.com. Google search domains are grouped under google.com, so clicks from Google AI Overviews count as google.com, while gemini.google.com stays separate. You can also connect Google Search Console to backfill 16 months of queries, clicks, impressions and CTR next to your traffic reports.

3. Measure conversions from AI visitors with goals

In Clicktag, you can set up to 50 goals and funnels per site (funnels have 2 to 6 steps), based on a pageview or a window.tt_event() custom event. Goals are computed from the data you already collect, so a new goal shows past conversions right away. Break a goal down by referrer or entry page to compare chatgpt.com and perplexity.ai visitors with google.com.

FAQ

Does blocking GPTBot or Google-Extended hurt my AI search rankings?

No. OpenAI documents that GPTBot (training) and OAI-SearchBot (search) settings are independent, and Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Just leave OAI-SearchBot and Googlebot allowed in robots.txt.

Why don't AI crawlers show up in normal web analytics tools?

Bots like OAI-SearchBot, ClaudeBot and PerplexityBot fetch the raw HTML of your page and don't run JavaScript, so client-side tracking scripts never fire. Clicktag reads their requests from your Cloudflare zone's analytics with a read-only token instead.

Why do some AI visits show up under Direct / none?

When an AI app sends neither a Referer header nor a utm_source parameter, no analytics tool can see where the click came from, so Clicktag groups it under Direct / none. Links that ChatGPT shows include utm_source=chatgpt.com, which Clicktag attributes to chatgpt.com.

Does Clicktag block bad AI bots or just report on them?

No, it only reports on crawler activity. The Crawlers tab's blocked count shows verified hits that got a 401, 403 or 429, so you can spot firewall rules turning search bots away.

What are Clicktag's limits?

Clicktag is free with no credit card required and supports up to 50 websites per account, up to 50 goals and funnels per site, and up to 12 filters at once. It doesn't include session recordings, heatmaps, bounce rate, revenue tracking or Google Analytics imports.