AI Crawler Directives: The Robots.txt Guide for GEO & AEO (2026)

Why Robots.txt Is Suddenly a GEO Problem, Not Just an SEO One

For twenty years, robots.txt existed to manage one relationship: your site and Googlebot. Get it wrong and you lost search rankings. That risk hasn’t gone away, but it’s no longer the only one.

Answer engines like ChatGPT, Perplexity, and Google’s AI Overviews now rely on a separate fleet of crawlers to decide what gets cited, quoted, and linked in an AI-generated answer. Many of these crawlers are distinct from the classic search bots your robots.txt already accounts for. A directive written in 2019 to control Googlebot’s crawl budget can silently block the exact bot that determines whether your product pages ever appear inside a ChatGPT or Perplexity answer.

This is the practical overlap between Technical SEO and Generative Engine Optimization (GEO) and Answer Engine Optimization (AEO): if the crawler can’t reach the page, no amount of schema markup, editorial quality, or query-fan-out optimization matters. Access is the precondition, not a nice-to-have.

The Current AI Crawler Landscape

Bot behavior and naming change as vendors update their crawling infrastructure, so treat the table below as a working reference to verify periodically rather than a permanent list.

Bot / User-Agent	Operator	Function	Blocks Citation If Disallowed?
GPTBot	OpenAI	Crawls content for training OpenAI’s generative foundation models. Not used for ChatGPT search or answers.	No — training only. Does not affect ChatGPT search citations.
OAI-SearchBot	OpenAI	Crawls and indexes pages specifically to surface them in ChatGPT’s search features.	Yes — disallowing removes the page from ChatGPT search answers (navigational links may still appear).
ChatGPT-User	OpenAI	Fetches a page in real time when a user asks ChatGPT or a Custom GPT to open a specific URL, or via GPT Actions. Not used for automatic crawling or search opt-outs.	Situational — blocks on-demand, user-triggered fetches only; OpenAI notes robots.txt rules may not apply since a user initiated the request.
ClaudeBot	Anthropic	Crawls content that may contribute to training Anthropic’s generative models.	No — training only. Disabling signals future content should be excluded from training data.
Claude-User	Anthropic	Fetches a page when a person asks Claude a question that requires visiting that specific site.	Yes — disabling prevents Claude from retrieving that content in response to a user query.
Claude-SearchBot	Anthropic	Crawls and indexes content specifically to improve Claude’s search result quality.	Yes — disabling reduces visibility and accuracy in Claude’s search results.
Google-Extended	Google	A robots.txt control token (not a separate crawler — uses existing Googlebot traffic) governing whether crawled content trains future Gemini models or is used for grounding in Gemini Apps / Vertex AI.	No effect on AI Overviews or standard Search — Google states it does not impact Search inclusion or ranking, and covers Gemini/Vertex AI training and grounding only.
PerplexityBot	Perplexity AI	Crawls and indexes pages to surface and link them in Perplexity’s search results. Not used to train AI foundation models.	Yes — directly removes the page as a citable search source.
Perplexity-User	Perplexity AI	Fetches a page when a user’s question requires visiting that specific site for an answer.	Limited effect — this fetcher generally ignores robots.txt since it’s a direct user request.
Applebot-Extended	Apple	A secondary token controlling whether Applebot-crawled content trains Apple’s generative foundation models (Apple Intelligence, etc.). Does not crawl pages itself.	No, not by itself — training opt-out only. Use the nosnippet meta tag to stop Apple’s AI answers from using a page’s content as context.
Bytespider	ByteDance	Crawls public content, reportedly for ByteDance/TikTok AI and search products.	Unclear — ByteDance has not published official crawler documentation, and third-party reports describe inconsistent robots.txt compliance.
CCBot	Common Crawl (non-profit)	Builds the open Common Crawl dataset, which many third-party LLMs use as training data.	Indirect — affects any model trained on Common Crawl’s dataset, not one specific platform.

Table 1: Major AI crawlers relevant to e-commerce sites, as of this writing. Verify current user-agent strings against each vendor’s published documentation before finalizing robots.txt changes.

Source Grading on This Table Bot names and stated purposes above are graded per Alneeko’s standard framework, and were checked directly against each operator’s live documentation as of this writing: VERIFIED — OpenAI, Anthropic, Google, Apple, and Common Crawl rows are confirmed against each operator’s own current, published crawler documentation. VENDOR CLAIM — where a page describes stated intent (e.g., “helps improve the capabilities of our models”) rather than independently testable behavior. DISPUTED — Bytespider. ByteDance has not published official crawler documentation for it, and third-party reports disagree on how consistently it honors robots.txt. Treat any “official” Bytespider spec you find secondhand with caution.

The Distinction That Actually Matters: Training vs. Retrieval

Most site owners lump every AI bot into one category — “AI scrapers” — and either block all of them out of caution or allow all of them without thinking it through. Neither is a strategy. The distinction that matters for e-commerce visibility is training versus retrieval.

Training crawlers

These tokens (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) govern whether crawled content feeds a model’s underlying training data. This affects a model’s general knowledge over time, not what it cites in a specific answer today. Blocking these is a brand and IP-control decision, not a visibility one — you can disallow GPTBot for training and still show up in ChatGPT’s live search results, because that runs through the separate OAI-SearchBot.

Google-Extended and Applebot-Extended are worth calling out specifically, because both are easy to over-credit with more power than they have. Neither is a crawler in its own right — both ride on their platform’s existing crawl traffic (Googlebot, Applebot) and act purely as a training/grounding opt-out flag. Per Google’s own documentation, Google-Extended has no effect on standard Search or on AI Overviews — it governs only Gemini Apps and Vertex AI training and grounding. Per Apple’s documentation, Applebot-Extended controls foundation-model training only; if the goal is to stop Apple’s AI answers (Siri, Apple Intelligence) from using a page as source context, that’s a job for the nosnippet meta tag, not Applebot-Extended.

Retrieval / answer crawlers

These bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) fetch and index pages specifically to surface them as live citations or answers. Blocking these directly removes your pages from consideration the next time someone asks an AI assistant a question your product page could answer. Note that the “-User” agents (ChatGPT-User, Claude-User, Perplexity-User) fire on a specific user’s request in the moment, and each vendor notes that robots.txt compliance for these can be partial, since a real person is waiting on that one fetch.

The practical takeaway: audit your robots.txt bot-by-bot, not with a single blanket rule. A blanket “Disallow: /” rule scoped broadly enough to catch “AI bots” often catches the retrieval crawlers you actually want, and a rule aimed at “Google-Extended” won’t touch Google AI Overviews at all — that’s a common, understandable mix-up worth double-checking before a client meeting.

Common Mistakes We See on E-Commerce Sites

• Legacy bot-management rules written for scraper protection years ago, using wildcard patterns that inadvertently match new AI user-agents.
• CDN or WAF-level bot blocking (Cloudflare, Akamai) that filters AI crawlers before they ever reach robots.txt — meaning the robots.txt file looks correct while the bot is still blocked at the network layer.
• Copy-pasted robots.txt from a template or a competitor, inheriting Disallow rules with no idea what they cover.
• Blocking all AI bots by default as a knee-jerk data-privacy move, without separating training exposure from citation opportunity.
• Assuming an llms.txt file substitutes for correct robots.txt access — it doesn’t. llms.txt is a curation layer that tells compatible AI systems what you’d like them to prioritize; it has no authority to grant crawl access a robots.txt Disallow has already blocked.
• Confusing a training opt-out with an answer opt-out — assuming Google-Extended or Applebot-Extended controls whether a page appears in AI Overviews or Siri’s AI answers, when both vendors state those tokens govern model training/grounding only.
• Treating Bytespider like the other bots on this list — ByteDance has not published official crawler documentation, and third-party monitoring reports inconsistent robots.txt compliance, so a Disallow line is not a guaranteed opt-out the way it is for OpenAI, Anthropic, or Google.

Writing the Directives: A Working Example

A reasonable baseline for an e-commerce site that wants citation visibility in AI answers while still controlling training use of its content might look like this. Treat it as a starting point to adapt, not a copy-paste fix — every site’s existing robots.txt has its own history and exceptions.

User-agent: OAI-SearchBot

Allow: /

User-agent: ChatGPT-User

Allow: /

User-agent: Claude-SearchBot

Allow: /

User-agent: Claude-User

Allow: /

User-agent: PerplexityBot

Allow: /

User-agent: GPTBot

Disallow: /   # optional — blocks training use only, does not affect ChatGPT search citations

User-agent: ClaudeBot

Disallow: /   # optional — blocks training use only, does not affect Claude-SearchBot/Claude-User

User-agent: Google-Extended

Disallow: /   # optional — blocks Gemini/Vertex AI training only; no effect on Search or AI Overviews

User-agent: Applebot-Extended

Disallow: /   # optional — blocks Apple foundation-model training only; use nosnippet for AI-answer opt-out

User-agent: CCBot

Disallow: /   # optional — same logic, affects any model trained on Common Crawl's dataset

Example only. The Disallow lines are a brand/training-control choice, not a technical requirement for citation visibility — some clients prefer to allow them for broader model presence. Confirm current directive support on each vendor’s crawler documentation page before publishing.

How This Connects to llms.txt and Schema

Robots.txt answers one question: can the crawler reach the page at all? It says nothing about whether the page is worth citing once reached. That’s where the rest of the technical stack does its job:

• llms.txt curates which sections of the site an AI system should treat as authoritative or prioritize when summarizing.
• Structured data (schema) gives the crawler machine-readable confirmation of what the page is — a product, a review, an FAQ — reducing ambiguity in how it gets represented in an answer.
• JavaScript rendering determines whether the retrieval crawler sees the same content a human browser does, or an empty shell — a gap Alneeko has flagged before on client audits and on our own site during initial buildout.

The same JS-heavy pages that hurt Core Web Vitals can just as easily hide content from AI crawlers, since both are ultimately parsing the same rendered DOM — a slow, render-blocking product page is failing two audiences with the same root cause.

Get the access layer wrong, and none of the rest matters — a beautifully marked-up product page an AI crawler can’t reach is invisible either way.

Auditing Your Own Site

A basic self-audit takes four steps:

(1) pull the live robots.txt at yourdomain.com/robots.txt and list every User-agent block against the crawler table above;

(2) check CDN/WAF bot-management settings separately, since robots.txt compliance is voluntary and a firewall rule can override it entirely;

(3) test a sample of key pages by fetching them as each retrieval bot’s user-agent to confirm the response isn’t a block page or CAPTCHA; and

(4) re-check quarterly, since crawler names and behaviors change faster than most technical SEO documentation does.

Alneeko Take This is a five-minute check with real downside if skipped. We’ve seen sites with excellent product content and clean schema that simply never show up in AI answers — because a WAF rule three years old is silently blocking every retrieval crawler that would have cited them.

Where This Fits an Audit Call

This is exactly the kind of check Alneeko runs at the start of every technical SEO and AI-readiness audit: cross-referencing robots.txt, CDN bot rules, and live crawler fetch tests against the current retrieval-bot landscape, before touching content or schema at all. If you haven’t checked whether your site is actually reachable by the bots feeding today’s AI answers, that’s the right place to start.

Sources

Every crawler entry in Table 1 was checked directly against the operator’s own current documentation on the date of writing. Readers are encouraged to verify independently, since crawler behavior and directive support change as vendors update their systems.

• OpenAI — Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
• Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• Google — Google’s common crawlers (crawler/fetcher reference, incl. Google-Extended): https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
• Perplexity — Perplexity Crawlers: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
• Apple — About Applebot: https://support.apple.com/en-us/119829
• Common Crawl — CCBot: https://commoncrawl.org/ccbot

No official ByteDance documentation for Bytespider was found at time of writing; its row in Table 1 is graded DISPUTED and based on third-party monitoring rather than a vendor source.