Crawler policy
Robots.txt for AI crawlers: four complete policies for four choices
Choose broad access, search without training, a documentation-only scope, or a decline for the listed AI agents—then test the delivered file.
Choose the publishing result first: broad access, search without listed training uses, documentation-only access for listed agents, or a decline for listed AI agents while ordinary search stays available. I would use one of the four complete files below as a starting policy, then reconcile it with the host’s existing rules.
Each example below is a complete file for an illustrative host. Replace example.com, adapt the paths and merge any existing requirements before deployment. Publishing all four together would combine conflicting choices. The AI crawler policy guide explains the provider map behind them.
Method & limits
RFC 9309 and provider documentation were checked October 21, 2026. The four files, hostnames, and expected parser results are illustrative. sulayman-bowles.dev’s deployed configuration and provider compliance remain outside the examples. Google’s parser CLI was documented but remains unbuilt here, with provider delivery untested. Private material needs authorization.
How do the groups match?
Consecutive User-agent lines share following rules, matching named groups supply the applicable policy, and repeated groups for one product combine before the wildcard fallback is considered, so RFC 9309 makes group selection a parsing decision rather than a last-setting-wins interpretation of the file.
Within the applicable rules, the most specific matching path wins, and an equivalent allow/disallow tie favors Allow. Product-token matching is case-insensitive; path matching should be case-sensitive. These examples use simple path prefixes so you can test the intended result without relying on an extension. RFC 9309, section 2.2.2.
That named-group rule creates a common mistake. If the wildcard excludes /search/ and a newly added OAI-SearchBot group only says Allow: /, the named bot no longer receives the wildcard exclusion. Either leave it on the fallback or carry the exclusions into its matching rules.
1. A public archive that permits broad crawler access
This fits an owner who has decided to permit ordinary search and the documented AI collection uses on its public archive. The exclusions avoid two unhelpful crawl spaces. They do not protect a secret: temporary exports containing private information should require authorization or should never be public.
No named provider group is needed here. The wildcard supplies the same default policy to the listed agents that follow it. Adding a long list of redundant allow groups would create more places to forget these exclusions.
Full illustrative robots.txt. Adapt the exclusions and replace the sitemap host.
User-agent: *
Allow: /
Disallow: /search/
Disallow: /exports/tmp/
Sitemap: https://example.com/sitemap.xml2. A portfolio or publisher that allows search and declines training
Search agents use the wildcard policy while GPTBot and ClaudeBot receive a site-wide disallow. OpenAI documents GPTBot’s model-development role, and Anthropic describes ClaudeBot’s. Google-Extended covers the uses in Google’s own reference.
PerplexityBot stays on the wildcard path because Perplexity describes it as a search crawler outside foundation-model training. A provider name alone would be a poor basis for putting it in the training block. Perplexity’s agent definitions.
Full illustrative file. Search agents receive /search/ and /preview/ exclusions through the wildcard fallback.
User-agent: *
Allow: /
Disallow: /search/
Disallow: /preview/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
Sitemap: https://example.com/sitemap.xml3. A product site that limits listed AI agents to documentation
Here ordinary search crawlers can access the public site, while the named AI search and user agents are directed to /docs/, /releases/ and /assets/. The longer Allow prefixes reopen those paths beneath Disallow: /. Training collection is declined.
Choose this when documentation is the intended public reference and other product paths change rapidly or add low-value crawl space. Match directories and supporting assets to the site. User-fetch exceptions still apply.
Full illustrative file. Use real canonical path prefixes and check required assets.
User-agent: *
Allow: /
Disallow: /search/
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /
Allow: /docs/
Allow: /releases/
Allow: /assets/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
Sitemap: https://example.com/sitemap.xml4. A publisher that declines the listed AI agents
This declines the named agents while leaving Googlebot and other ordinary search crawlers on the wildcard policy. It is a maintained list, so a newly introduced token needs review. User-triggered fetchers and uncooperative clients require further attention.
The file also leaves Google’s generative Search presentation decision unresolved. Google-Extended does not control AI Overviews; Google now documents a Search generative AI setting in Search Console. Set that separately if the publisher wants to decline the specified Search features. Google’s control documentation.
Full illustrative file. This expresses a preference for user fetchers even where their documented behavior limits robots applicability.
User-agent: *
Allow: /
Disallow: /search/
User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
Disallow: /
Sitemap: https://example.com/sitemap.xmlWhich requests can these files fail to stop?
OpenAI describes exceptions for ChatGPT-User, Perplexity says its user fetcher generally ignores robots, and Anthropic says its bots honor the file. Those differences govern the consequences of the publisher’s user-fetch preferences.
For a technical deny, use an appropriate edge or application control and verify the response. For private material, require authorization. The robots access-control boundary explains why naming a path in this public file cannot protect it.
How do you validate the file you publish?
After deployment, save the public /robots.txt bytes for each host. A CDN or hosting setting can alter the origin file. Test the returned policy on an allowed page, a prohibited path, and every relevant named token.
Google’s open-source parser provides a documented single-URL CLI. After building it using the repository README, run the following from its checkout with an absolute path to your saved file. This checks parser interpretation, not whether another provider will make a request or honor the result.
Keep the served file, parser/version, and path decisions together, then check document delivery and attributed visits through the log method. I would accept the policy when the delivered rules express the chosen publication, including the exclusions it still needs.
| Example | Token and path | Expected |
|---|---|---|
| 1. Broad archive | GPTBot /papers/result | Allowed |
| 2. Search, no training | GPTBot /article | Disallowed |
| 2. Search, no training | OAI-SearchBot /article | Allowed |
| 2. Search, no training | OAI-SearchBot /search/results | Disallowed |
| 3. Documentation | OAI-SearchBot /docs/start | Allowed |
| 3. Documentation | OAI-SearchBot /pricing/ | Disallowed |
| 4. Decline listed AI | PerplexityBot /article | Disallowed |
| 4. Decline listed AI | Googlebot /article | Allowed |
Requires the documented Bazel build environment. These are commands to run, not results observed in this article.
bazel run robots_main -- /absolute/path/robots.txt OAI-SearchBot https://example.com/docs/start
bazel run robots_main -- /absolute/path/robots.txt OAI-SearchBot https://example.com/pricing/Sources
- RFC 9309 — Robots Exclusion ProtocolIETF / RFC Editor · September 2022Verified Oct 21, 2026
Grouping, fallback, path matching and access-authorization boundary checked.
- OpenAI — Overview of OpenAI CrawlersOpenAIVerified Oct 21, 2026
Official documentation tool fetch; no displayed publication date.
- Anthropic — web crawlers and site-owner controlsAnthropic · April 7, 2026Verified Oct 21, 2026
- Perplexity — Perplexity CrawlersPerplexityVerified Oct 21, 2026
No displayed publication date. Provider search/training and user-fetch descriptions checked.
- Google — Common crawlers / Google-ExtendedGoogle · Updated October 15, 2026Verified Oct 21, 2026
- Google Search Console Help — Search generative AI controlGoogleVerified Oct 21, 2026
Official page states worldwide rollout as of August 31, 2026; that is an event date, not a publication date.
- Google — robots.txt parser and matcher READMEGoogleVerified Oct 21, 2026
Single-URL Bazel CLI and build prerequisites checked. No parser build performed.