1. [Home](https://www.void-agency.com/)
2. [Insights](https://www.void-agency.com/insights)
3. AI Crawler Controls: Search, Training, and User Fetches

Technical SEO15 min readJuly 17, 2026

# AI Crawler Controls in 2026: Separate Search Visibility, Model Training, and User-Directed Fetches

Compare OpenAI, Google, Anthropic, and Perplexity crawler controls for search, training, user fetches, robots.txt, WAFs, IP ranges, and logs.

Written by**Sulayman Bowles**Field note 02

Crawler policy / three decisions

01**Search visibility** Search crawler

02**Model training** Training crawler

03**User-directed fetch** User action

**Search, training, and user fetches are separate lanes**Policy · delivery · verification

Crawler policy and verification

7

Evidence sections

13

Linked references

In this field note
1. 01Start with three read decisions—and one action boundary
2. 02OpenAI: OAI-SearchBot is not GPTBot
3. 03Google: Search access and Google-Extended have different effects
4. 04Anthropic and Perplexity also separate purposes
5. 05Write a page-class policy before writing robots rules
6. 06Verify the live path, not just the file
7. 07Measure the consequence of a policy change
8. RSource ledger

“Block AI bots” is no longer a coherent technical instruction. The same provider may operate one crawler for search discovery, another for possible model training, and a user-triggered agent that retrieves a page because a person asked it to. Those requests can have different robots behavior, different identity evidence, and different consequences for visibility.

A useful policy begins with purpose. Decide whether a page should be eligible for search answers, whether its content may be considered for model training, whether a user-directed assistant may retrieve it, and whether an authenticated agent may complete an action. Then map each decision to documented controls and verify what your origin actually serves. Purpose is not network identity: a provider may reuse a crawl or stored copy across documented uses, and a verified request proves a fetch—not every downstream use of the bytes.

Short answer

Write crawler policy by use case, not by the word “AI.” OpenAI, Google, Anthropic, and Perplexity document separate mechanisms for search, training, and user-triggered access. Use robots.txt for documented crawl preferences, use meta controls where a platform supports result removal or snippet limits, enforce real access restrictions at the application layer, and validate behavior in server and WAF logs. Recheck every provider's documentation before deployment: crawler names, identity evidence, and support windows can change, and Google's previous NotebookLM token reached the end of its documented transition window in August 2026.

Section 01

## Start with three read decisions—and one action boundary

Search inclusion, model training, and user-directed retrieval are three separate read purposes. A service business may want public service pages discoverable in ChatGPT Search while signaling that those pages should not be used for training OpenAI foundation models. A publisher may allow Google Search, including AI Overviews and AI Mode, while using Google-Extended to control specified Gemini training and grounding uses outside Search. A private portal should rely on authentication rather than assuming a robots rule makes the content private. State-changing agent actions are a fourth, higher-risk boundary governed by application security rather than crawler policy.

The policy questions that must not be collapsed

| Purpose | Business question | Typical control layer |
| --- | --- | --- |
| Search discovery | May this page be indexed or surfaced as a supporting result? | Provider crawler rule, indexability, and result-preview controls |
| Model training | May crawled content be considered for future foundation-model training? | Provider-specific training token or documented opt-out |
| User-directed retrieval | May an assistant fetch this page in response to a person's request? | Provider agent rule where honored, plus real access controls |
| Agent action | May software submit, book, buy, or change state? | Authentication, authorization, CSRF protection, confirmation, and audit logs |

Evidence boundary

RFC 9309 states that robots rules are not access authorization. A disallow directive communicates a crawl preference to compliant automated clients; it does not make a URL confidential or technically unreachable.

If content must be private, enforce authentication and authorization at the origin. Do not place secrets in a public URL and depend on robots.txt.

Crawl blocking is not the same as disappearance. Depending on the provider and other discovery signals, a blocked URL, title, domain, or limited description may still be known or shown. Use the provider's documented removal or preview controls, authentication, or deletion workflow for the outcome actually required.

Section 02

## OpenAI: OAI-SearchBot is not GPTBot

OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT search features. Allowing the user agent and requests from its published IP ranges helps make a site eligible; it does not guarantee placement. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although navigational links can still appear. OpenAI says its search systems can take approximately 24 hours to adjust to a robots change; that is not a guaranteed propagation deadline.

GPTBot has a different purpose. OpenAI says it crawls content that may be used to make generative AI foundation models more useful and safe, and that disallowing GPTBot indicates the site's content should not be used in training those models. The settings are independent: allowing OAI-SearchBot while disallowing GPTBot is a documented configuration, not a workaround.

ChatGPT-User supports certain actions initiated by a person in ChatGPT or a Custom GPT. OpenAI says it is not an automatic web crawler, is not used to determine Search inclusion, and may not be governed by robots rules because the request is user initiated. It publishes a separate IP-range endpoint for this agent. A WAF policy that allows OAI-SearchBot but blocks every browser-like or user-triggered request may therefore produce a different result from the robots policy.

### OpenAI verification

- Inspect the live robots response for OAI-SearchBot and GPTBot separately.
- Confirm that CDN and WAF rules do not contradict the intended robots policy.
- Validate source IPs against OpenAI's current published JSON ranges rather than trusting the user-agent string alone.
- Check representative public pages for successful status codes and usable HTML.
- Wait through the documented adjustment period before repeating inclusion tests.
- Treat observed ChatGPT links and tagged referrals as separate evidence from crawler access.

Section 03

## Google: Search access and Google-Extended have different effects

Google's AI Overviews and AI Mode are features of Google Search. Google says Googlebot is the site-owner control for how content is crawled for Search, including those generative AI features. Blocking Googlebot prevents recrawling and can remove the content and snippet eligibility needed for ordinary and generative Search supporting links. A URL discovered elsewhere can still be known or appear without a useful snippet, so crawl blocking is not a guaranteed URL-removal control.

Google-Extended is a standalone robots product token, not a separate HTTP user agent. Google says it controls whether content Google crawls may be used to train future Gemini models that power Gemini Apps and the Vertex AI Gemini API, and for grounding in those specified products. Google explicitly states that Google-Extended does not affect inclusion or ranking in Google Search.

Search-result presentation has additional controls. Google documents noindex for removal and nosnippet, data-nosnippet, and max-snippet for limiting information shown from a page. Those directives have different scopes from a crawler disallow. A crawler must be able to fetch a page to see a meta directive, which is why combining a robots block with a meta instruction can prevent the instruction from being read. When a specific user-agent group matches, it does not inherit rules from the wildcard group; groups for the same specific token may be combined. Restate every intended rule in the matching group and test it with Google's documented robots parser semantics.

Google also documents user-triggered fetchers that are distinct from Googlebot. Current examples include Google-Agent and Google-GeminiNotebook. Google's documentation identifies Google-NotebookLM as the previous NotebookLM user agent and says that previous token was supported through August 2026. For a September 2026 policy, treat Google-GeminiNotebook as the current NotebookLM token and do not rely on the expired transition window as evidence that Google-NotebookLM remains a supported equivalent. User-triggered fetchers generally ignore robots.txt because a person initiated the request. Some Google-Agent traffic can additionally carry signed Web Bot Auth evidence. None of those identities should receive privileged application access merely because its user-agent string looks familiar.

Do not infer a universal Gemini control

Google-Extended has the product effects described in Google's current crawler documentation. Do not describe it as a switch for every Google AI product, every user-triggered agent, or Google Search.

Google's common-crawler and user-triggered-fetcher documentation was rechecked on September 21, 2026. Recheck the affected-products language and current agent tokens before changing production policy.

September 2026 migration check

Search production configuration for the literal Google-NotebookLM token across robots.txt generators, WAF/CDN rules, allowlists, bot-management products, log parsers, dashboards, and alerting. If the business policy permits NotebookLM user-directed retrieval, update exact-token rules to Google-GeminiNotebook and preserve the intended application-layer access boundary.

Do not infer that a familiar user-agent string proves identity. Keep provider-documented network or cryptographic verification where available, record the source URL and review date, and use application authentication for privileged content or actions.

Section 04

## Anthropic and Perplexity also separate purposes

Anthropic documents three robots. ClaudeBot collects public web content that could contribute to model training. Claude-SearchBot navigates the web to improve search-result quality. Claude-User retrieves content at a user's direction. Anthropic says disabling Claude-SearchBot may reduce a site's visibility and accuracy in user search results, while disabling Claude-User may reduce visibility for user-directed web search. It says its bots honor robots rules and documents a non-standard Crawl-delay extension.

Anthropic's current crawler guidance links to an official bots.json file containing IP prefixes. It warns that blocking those addresses can prevent its bots from reading robots.txt and therefore may not guarantee the intended opt-out. Use the list as dated identity evidence, not a policy substitute: keep explicit robots rules as the documented crawl control, record when the list was retrieved, and recheck it before changing a WAF allowlist.

Perplexity documents PerplexityBot as its search-indexing crawler and Perplexity-User as a user-action agent. Its crawler documentation recommends allowing PerplexityBot and published bot IP ranges for search visibility, and says Perplexity-User generally ignores robots rules because a user requested the fetch. On July 16, 2026, Perplexity separately stated that direct URL summarization for a robots-blocked page had been disabled and that PerplexityBot crawls in compliance with robots.txt. Read those claims narrowly: they establish the stated search-crawler behavior and a disabled direct-summary path, not a universal promise that every user-directed or indirectly discovered workflow obeys the same rule.

Documented provider controls reviewed September 21, 2026

| Provider | Automated search crawler | Training-related use or control | User-triggered fetcher | Identity evidence / limitation |
| --- | --- | --- | --- | --- |
| OpenAI | OAI-SearchBot | GPTBot opt-out for stated foundation-model training | ChatGPT-User; robots may not govern user-initiated requests | Published IP JSON; a blocked crawler can still leave limited navigational links |
| Google | Googlebot for Google Search features | Google-Extended for specified Gemini training and grounding uses | Google-Agent and Google-GeminiNotebook; user-triggered fetchers generally ignore robots | DNS/IP verification plus Web Bot Auth for some Google-Agent requests; no universal Gemini switch |
| Anthropic | Claude-SearchBot | ClaudeBot training control | Claude-User; provider says robots rules are honored | Official bots.json is dated evidence, not a permanent allowlist |
| Perplexity | PerplexityBot; provider states robots compliance | Provider says PerplexityBot is not for foundation-model training | Perplexity-User; crawler docs say it generally ignores robots | Published IP ranges; direct blocked-URL summary statement does not settle every indirect path |

Training-control scope

A current opt-out can express how the provider says future crawling will be used. It does not prove that earlier copies were deleted, existing model weights changed, third-party datasets were purged, or every downstream product inherited the preference.

Report the exact token, product scope, deployment date, and provider wording. Do not translate a crawl preference into a broader data-deletion claim.

Section 05

## Write a page-class policy before writing robots rules

A single site-wide rule is rarely the best starting point. Public service pages, original research, account pages, staging environments, licensed media, and lead-confirmation pages have different business and privacy requirements. Classify those surfaces first. A public article may be deliberately available to search crawlers. A customer portal must be authenticated. A thank-you page may be public at the transport layer but should usually be noindexed and free of personal data.

### Policy design sequence

1. Inventory page classes and identify the owner of each class.
2. Decide the desired outcome for search discovery, model training, user retrieval, and agent action.
3. Map each outcome only to controls the provider officially documents.
4. Identify where authentication, authorization, or application logic is required instead of robots preferences.
5. Write the smallest explicit rules that implement the approved decisions.
6. Review CDN, WAF, bot-management, and hosting defaults for contradictory enforcement.
7. Record the policy date, source URLs, approver, and next review date.

Avoid unrecognized bot names copied from old lists. Providers rename products, introduce new agents, and change documentation. An unrecognized token can create the appearance of control while doing nothing. Conversely, a wildcard block may catch legitimate search crawlers and remove visibility. Every token in production policy should have a current first-party reference and a stated owner.

Crawler-policy drift register

| Signal | What to record | Release action |
| --- | --- | --- |
| Renamed or retired token | Previous token, current token, provider source, and documented support-end date | Update exact-match rules and regression tests; do not keep an expired alias by assumption |
| IP or verification change | Published range, DNS, or Web Bot Auth source plus retrieval date | Refresh network checks without treating them as authorization |
| Purpose or scope change | Search, training, user-fetch, or action scope in the provider wording | Re-evaluate the business decision before changing robots or WAF policy |
| Production mismatch | robots response, WAF result, origin status, and verified log identity | Fix the enforcement layer that contradicts the approved policy |

### Quarterly crawler-policy review

- Reopen every first-party provider source referenced by production rules.
- Diff current crawler and user-agent names against robots, WAF, CDN, bot-management, and analytics configuration.
- Remove assumptions whose documented transition or support window has expired.
- Re-run representative requests and verify status, rendered content, and origin or WAF logs.
- Record owner, evidence date, approved policy decision, and next review date.

robots.txt delivery states matter

RFC 9309 distinguishes an unavailable robots file from an unreachable one. A 4xx response can permit crawling; a 5xx or network-unreachable state requires compliant clients to assume complete disallow. A cached robots file should normally be reused for no more than 24 hours unless the file remains unreachable.

Monitor status, caching, redirects, and body—not only rule syntax. A WAF or outage that changes the robots response can change the effective policy without a repository edit.

Section 06

## Verify the live path, not just the file

A correct robots file can coexist with a broken delivery path. A CDN may serve an old cached file. A WAF may challenge the bot before it reaches the origin. A hosting rule may return different content by user agent. JavaScript may hide the important text from a crawler that does not execute the same rendering path. Verification therefore needs requests, responses, and logs.

### Production validation gate

- Fetch `/robots.txt` from the public canonical host and record status, body, headers, and timestamp.
- Run a synthetic parser or request test with the documented token and label it as configuration evidence, not provider traffic.
- Where providers publish ranges, compare the observed source address with the current official JSON endpoint.
- Inspect WAF events for challenges, managed-bot decisions, rate limits, and false positives.
- Confirm response status, canonical, robots meta directives, and important textual content.
- Check server logs after the provider's stated propagation window and label only requests with verified provider identity as real crawler traffic.
- Do not search logs for a Google-Extended HTTP user agent; Google-Extended is a robots product token, not a request identity.
- Run a separate search-result or citation observation; crawler success alone does not prove visibility.
- Retain evidence without storing secrets or unnecessary personal request data.

Identity is a security question

User-agent strings are self-declared and can be spoofed. OpenAI, Perplexity, and Google publish verification material; Google also documents reverse/forward DNS and experimental Web Bot Auth for some requests.

Do not grant privileged access solely because a request claims to be a reputable AI agent.

Section 07

## Measure the consequence of a policy change

A crawler-policy release should have a change record and an observation window. Capture the old and new robots files, affected page classes, provider documentation, deployment time, CDN purge state, and expected propagation. Then inspect crawler requests, indexability, observed search links, referrals, and lead outcomes as separate measures.

Do not attribute every visibility change to the robots edit. Search systems recrawl on their own schedules, generated answers vary, and provider products change. A controlled audit can show that a crawler became allowed and later fetched the page. It cannot prove that the rule caused a citation unless the design includes adequate controls and repeated observations.

Crawler policy change ledger

| Evidence layer | Record | Claim allowed |
| --- | --- | --- |
| Configuration | Versioned robots and WAF rules | The intended policy changed |
| Transport | Dated successful or blocked requests | The tested request received this response |
| Crawler activity | Verified log entries | A documented crawler accessed the URL |
| Search observation | Captured result, prompt, date, and surface | The page appeared in that observation |
| Indirect discovery | Dated link, title, snippet, or third-party source observed without a verified direct fetch | The system knew or displayed limited information; direct crawl is not established |
| Referral | Tagged or attributed landing session | A visit arrived with that source evidence |
| Outcome | Accepted lead or revenue record | A business outcome occurred under the stated attribution limits |

Volatile as of September 21, 2026

Provider agent names, version strings, IP ranges, products, propagation times, and robots behavior can change. Google completed the documented transition window for the previous NotebookLM user-agent token after this article was first published.

Re-open every first-party crawler reference before implementing or renewing a policy. A dated template or allowlist is not evidence of current behavior.

Primary and authoritative references

## Source ledger

These sources support the operating guidance above. Platform behavior and documentation can change, so volatile implementation details should be rechecked before a rollout.

1. [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) — OpenAI. Purpose, independence, agent strings, propagation guidance, and published IP endpoints for OpenAI agents.
2. [ChatGPT Search](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt) — OpenAI Help Center. Search eligibility, possible citations, crawler access and published IP ranges; placement is not guaranteed. The crawler reference above documents the distinct agents and navigational-link behavior.
3. [List of Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) — Google Crawling Infrastructure. Googlebot and Google-Extended effects; last updated July 14, 2026.
4. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Google Search Central. Search eligibility and preview controls for AI Overviews and AI Mode.
5. [List of Google's user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) — Google Crawling Infrastructure. Current user-triggered agent tokens, robots behavior, and product-name transition.
6. [Verifying Googlebot and other Google crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests?hl=en) — Google Crawling Infrastructure. DNS and published-IP verification methods for Google crawler traffic.
7. [Web Bot Auth](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) — Google Crawling Infrastructure. Signed identity evidence available for some Google-Agent requests.
8. [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) — Google Search Central. Matching groups, wildcard behavior, rule combination, and Google-specific parser semantics.
9. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic Help Center. ClaudeBot, Claude-SearchBot, Claude-User, robots behavior, and the linked bots.json IP-prefix list with maintenance caveats.
10. [Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) — Perplexity. PerplexityBot, Perplexity-User, published IP ranges, and WAF guidance.
11. [How does Perplexity follow robots.txt?](https://www.perplexity.ai/help-center/en/articles/10354969-how-does-perplexity-follow-robots-txt) — Perplexity Help Center. July 16, 2026 statement about current robots behavior and prior URL-summary behavior.
12. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) — IETF. Standards-track robots syntax, access behavior, caching, and explicit non-authorization boundary.
13. [Crawler documentation changelog](https://developers.google.com/crawling/docs/changelog) — Google Crawling Infrastructure. Current crawler and user-triggered fetcher documentation changes, including the NotebookLM token transition.

Implementation

## The practical next step

Inventory five representative page classes, decide separately whether each should support search discovery, model training, user-directed retrieval, and agent actions, then test every provider-specific robots group and access rule at the public origin. Preserve direct-fetch, identity, indirect-discovery, and result evidence as separate layers, and re-open every provider reference before deployment.

[Measure crawler access separately from mentions, citations, and referrals](https://www.void-agency.com/insights/measure-ai-search-visibility)[Review the broader AI-search readiness checklist](https://www.void-agency.com/insights/austin-ai-search-readiness)[Check whether delivery and page performance obstruct real users](https://www.void-agency.com/insights/site-speed)

## Related service pages

[AI, website, and SEO systems for service businesses in Austin](https://www.void-agency.com/smbs)

### More Field Notes

## Related Reading

Technical SEO

### Googlebot’s 2 MB HTML Limit: A Byte-Level Audit

Test what Googlebot can fetch before the 2 MB cutoff and whether critical metadata, links, content, ...

[Read more](https://www.void-agency.com/insights/googlebot-2mb-html-limit-audit)AI Search

### Google AEO/GEO Claims in 2026

Google now rejects special ranking promises attached to several popular AEO and GEO tactics without ...

Read more Technical SEO

### SPA Core Web Vitals in Chrome 151: Soft Navigations

New browser support makes route-transition performance more observable, but teams still need feature...

Read more

---
Source: https://www.void-agency.com/insights/ai-crawler-controls-search-training-user-fetch · VOID Agency
