Can AI assistants reach your website? Crawlers, robots.txt, and access

One provider, several jobs

OpenAI makes the split unusually clear. OAI-SearchBot supports ChatGPT Search discovery, GPTBot collects material that may contribute to foundation-model development, ChatGPT-User visits at a user’s request, and OAI-AdsBot validates submitted ad landing pages. OpenAI says its ad-validator collection is not used for foundation-model training. Blocking that validator can create an ad-review problem without changing the training preference at all. OpenAI crawler documentation.

The providers disagree about user-directed fetches: Anthropic says Claude-User honors robots, while Perplexity says Perplexity-User generally ignores them. “User” names a workflow, not a common permission contract. An incident needs that product’s documented policy and the request evidence.

A publisher can decline all documented automated uses together when maintaining product-specific permissions costs more than their distinctions are worth, but that simpler intention still needs to account for the receiving products’ different user-fetch exceptions before the owner can judge which requests fit the policy.

The policy file can express the choice

The first template allows search-oriented access and blocks the two documented model-development crawlers. The second declines access for all eight identities in the original policy sheet. They are starting policies, and both must be reconciled with the file already serving the host.

A named OAI-SearchBot group containing only Allow: / can reopen /internal-search/ and /facets/ when those blocks live only under *, because RFC 9309 selects matching product groups before its fallback and gives them their own rules to apply to the requested path. Copy the intended exclusions and test both an article and an excluded URL. Product tokens match without case sensitivity; paths should match case-sensitively. The RFC interpretation is the rule to check when adding a named group.

Check each host’s public policy bytes and cache state against the intended file. That verifies delivery. To observe access, keep an attributed page request and its response; to judge discovery, measure what follows.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Sitemap: https://example.com/sitemap.xml

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

Google-Extended never appears in the request log

Google-Extended is a robots product token for specified Gemini training and grounding uses. Google says ordinary agents make the requests, leaving logs without a dedicated Google-Extended User-Agent to count. Record the use preference in the policy and test the network request under its ordinary identity.

Google-Extended leaves Search inclusion and ranking unchanged. Google’s Search AI guidance now describes a Search Console setting for AI Overviews, AI Mode, and generative features in Discover, rolled out worldwide on August 31, 2026. Record that property setting alongside training preferences and page-level noindex or snippet controls. Googlebot must be able to read applicable page directives.

Google’s product-token description exposes the limit of a request-centered audit. Google-Extended governs documented downstream uses without creating a network identity to count. Logs can show a verified Google request at the measured point. Testing the downstream choice requires more than searching for a User-Agent that doesn’t exist.

User-agent: Google-Extended
Disallow: /

A user request changes the rules

A Perplexity-User visit after a PerplexityBot disallow can fit the documented user-request exception. The same explanation doesn’t transfer to Claude-User, whose provider says it honors robots. Judge a request against its exact product and the policy served at that time.

The model-development choice concerns future collection. Restricting ClaudeBot expresses that preference; it supplies no account of material Anthropic already holds. That limitation belongs beside the policy decision, because a new rule cannot describe the history of previously collected copies.

For a public essay, decide whether blocking automatic collection while allowing assisted retrieval fits the publication’s purpose. Private content needs an enforced permission check regardless of the requester’s name, as RFC 9309 makes clear.

An allowed bot can still receive a 403

Imagine a public article that robots.txt allows OAI-SearchBot to fetch, followed by an attributed request that reaches the CDN and receives 403 from a WAF rule. Editing the GPTBot group will not fix it. Preserve the denied request, locate the matching edge rule and check the response after a scoped repair. A successful OAI-AdsBot visit would concern a different workflow.

The shell filter below finds claimed identities in current and rotated NGINX logs. A copied User-Agent is cheap. Attribution requires the trusted client address and the corresponding provider-published network evidence, with the exact range snapshot saved beside the classification. OpenAI and Perplexity publish product-specific feeds; Google publishes IP and DNS verification methods; Anthropic supplies bot-network guidance. Verification method.

For a release check I would save the public robots file, parser decisions for representative paths, and any attributed article response. OpenAI’s September guidance allowed roughly 24 hours for Search policy updates. If the request window stays empty, record provider delivery as unobserved: the same silence can follow a new policy or a crawler with nothing to fetch.

zgrep -hEi 'OAI-SearchBot|GPTBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' /var/log/nginx/access.log*

curl -sS -D - https://example.com/robots.txt -o /dev/null

The registry is part of the release

The registry should preserve the product, documented purpose, request mode, chosen policy, official verification source, and date. Attach network snapshots to classifications and keep provider promises attributable to their documents, so a later policy update can change the promise without silently rewriting an earlier observed response.

A feed outage remains an unknown input. Replacing it with an empty list would turn missing evidence into a collection of apparent failed matches. A retained snapshot can support a classification under a declared age policy; it cannot become contemporaneous merely because it is the only file available.

A small publication may need a versioned policy and release checklist. A traffic system earns its cost when recurring disputes or product changes demand request history. I would require a proposed diff, parser decisions for representative paths, and public readback before calling a rule deployed. A verified request and response can then show access. Search inclusion, citations, and training use need their own measurements.

One page can locate the failed control

Choose a public page with a useful sentence, a link and any data the reader needs. Save the final robots.txt response and evaluate the rule for that exact path. Then save the page’s redirect chain, status, content type and body. A 200 challenge or login page is a different outcome from the article.

Compare the initial text with the browser through the HTML and rendered-page method. In logs, classify claimed names and verified networks independently. A laptop request wearing a bot label is still a laptop probe.

Keep a dated result for the URL, bot product, policy decision, response, and content. Repair the point that failed and repeat that check. I would call access working when the intended caller receives the intended public page, then measure what the assistant does with it.

Use the page-access checklist to test delivery, then weigh the decision to block GPTBot against the use you want to permit.