AI Crawler Controls in 2026: Separate Search Visibility, Model Training, and User-Directed Fetches

Search inclusion, model training, and user-directed retrieval are three separate read purposes. A service business may want public service pages discoverable in ChatGPT Search while signaling that those pages should not be used for training OpenAI foundation models. A publisher may allow Google Search, including AI Overviews and AI Mode, while using Google-Extended to control specified Gemini training and grounding uses outside Search. A private portal should rely on authentication rather than assuming a robots rule makes the content private. State-changing agent actions are a fourth, higher-risk boundary governed by application security rather than crawler policy.

OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT search features. It recommends allowing the user agent and requests from its published IP ranges to help ensure inclusion. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although navigational links can still appear in specified circumstances. A robots change can take approximately 24 hours to propagate through OpenAI's search systems.

GPTBot has a different purpose. OpenAI says it crawls content that may be used to make generative AI foundation models more useful and safe, and that disallowing GPTBot indicates the site's content should not be used in training those models. The settings are independent: allowing OAI-SearchBot while disallowing GPTBot is a documented configuration, not a workaround.

ChatGPT-User supports certain actions initiated by a person in ChatGPT or a Custom GPT. OpenAI says it is not an automatic web crawler, is not used to determine Search inclusion, and may not be governed by robots rules because the request is user initiated. It publishes a separate IP-range endpoint for this agent. A WAF policy that allows OAI-SearchBot but blocks every browser-like or user-triggered request may therefore produce a different result from the robots policy.

Google's AI Overviews and AI Mode are features of Google Search. Google says Googlebot is the site-owner control for how content is crawled for Search, including those generative AI features. Blocking Googlebot prevents recrawling and can remove the content and snippet eligibility needed for ordinary and generative Search supporting links. A URL discovered elsewhere can still be known or appear without a useful snippet, so crawl blocking is not a guaranteed URL-removal control.

Google-Extended is a standalone robots product token, not a separate HTTP user agent. Google says it controls whether content Google crawls may be used to train future Gemini models that power Gemini Apps and the Vertex AI Gemini API, and for grounding in those specified products. Google explicitly states that Google-Extended does not affect inclusion or ranking in Google Search.

Search-result presentation has additional controls. Google documents noindex for removal and nosnippet, data-nosnippet, and max-snippet for limiting information shown from a page. Those directives have different scopes from a crawler disallow. A crawler must be able to fetch a page to see a meta directive, which is why combining a robots block with a meta instruction can prevent the instruction from being read. When a specific user-agent group matches, it does not inherit rules from the wildcard group; groups for the same specific token may be combined. Restate every intended rule in the matching group and test it with Google's documented robots parser semantics.

Google also documents user-triggered fetchers that are distinct from Googlebot. Current examples include Google-Agent and Google-GeminiNotebook; Google-NotebookLM remains the former product token during a documented transition through August 2026. Google says user-triggered fetchers generally ignore robots.txt because a person initiated the request. Some Google-Agent traffic can additionally carry signed Web Bot Auth evidence. None of those identities should receive privileged application access merely because its user-agent string looks familiar.

Anthropic documents three robots. ClaudeBot collects public web content that could contribute to model training. Claude-SearchBot navigates the web to improve search-result quality. Claude-User retrieves content at a user's direction. Anthropic says disabling Claude-SearchBot may reduce a site's visibility and accuracy in user search results, while disabling Claude-User may reduce visibility for user-directed web search. It says its bots honor robots rules and documents a non-standard Crawl-delay extension.

Anthropic's current crawler guidance links to an official bots.json file containing IP prefixes. It warns that blocking those addresses can prevent its bots from reading robots.txt and therefore may not guarantee the intended opt-out. Use the list as dated identity evidence, not a policy substitute: keep explicit robots rules as the documented crawl control, record when the list was retrieved, and recheck it before changing a WAF allowlist.

Perplexity documents PerplexityBot as its search-indexing crawler and Perplexity-User as a user-action agent. Its crawler documentation recommends allowing PerplexityBot and published bot IP ranges for search visibility, and says Perplexity-User generally ignores robots rules because a user requested the fetch. On July 16, 2026, Perplexity separately stated that direct URL summarization for a robots-blocked page had been disabled and that PerplexityBot crawls in compliance with robots.txt. Read those claims narrowly: they establish the stated search-crawler behavior and a disabled direct-summary path, not a universal promise that every user-directed or indirectly discovered workflow obeys the same rule.

A single site-wide rule is rarely the best starting point. Public service pages, original research, account pages, staging environments, licensed media, and lead-confirmation pages have different business and privacy requirements. Classify those surfaces first. A public article may be deliberately available to search crawlers. A customer portal must be authenticated. A thank-you page may be public at the transport layer but should usually be noindexed and free of personal data.

Avoid unrecognized bot names copied from old lists. Providers rename products, introduce new agents, and change documentation. An unrecognized token can create the appearance of control while doing nothing. Conversely, a wildcard block may catch legitimate search crawlers and remove visibility. Every token in production policy should have a current first-party reference and a stated owner.

A correct robots file can coexist with a broken delivery path. A CDN may serve an old cached file. A WAF may challenge the bot before it reaches the origin. A hosting rule may return different content by user agent. JavaScript may hide the important text from a crawler that does not execute the same rendering path. Verification therefore needs requests, responses, and logs.

A crawler-policy release should have a change record and an observation window. Capture the old and new robots files, affected page classes, provider documentation, deployment time, CDN purge state, and expected propagation. Then inspect crawler requests, indexability, observed search links, referrals, and lead outcomes as separate measures.

Do not attribute every visibility change to the robots edit. Search systems recrawl on their own schedules, generated answers vary, and provider products change. A controlled audit can show that a crawler became allowed and later fetched the page. It cannot prove that the rule caused a citation unless the design includes adequate controls and repeated observations.