Crawler policy
Robots.txt crawl-delay: Google ignores it, Bing and Claude support it
Crawl-delay is an extension some providers honor. Your server still owns the amount of work it accepts.
Google ignores Crawl-delay. Bing documents support, Anthropic supports the non-standard extension, and Yandex stopped honoring it in 2018. Check the caller before using the directive to slow requests: publishing a delay only works when that client agrees to wait.
The comparison below gives each provider’s documented behavior. A local server and synthetic client then demonstrate the difference between requesting a delay and enforcing one.
Method & limits
Provider documentation checked October 26, 2026. The response designs and local-host demonstration are authored examples. Behavior by search engines and AI providers was not measured.
The same line produces different expectations
RFC 9309 permits extra records without adding them to its standard rule set. A crawler can implement Allow and Disallow correctly while ignoring Crawl-delay. Verify which product token reaches your host, then apply that provider’s pacing policy.
| Provider / crawler | Crawl-delay support | Practical choice |
|---|---|---|
| Googlebot | Not supported by Google’s robots parser. | Investigate crawl traps and load. Use documented temporary capacity responses when necessary. |
| Bingbot | Supported. Bing describes a one-URL-per-window interpretation. | Use an explicit Bingbot group; the delay takes precedence over Bing Webmaster Crawl Control. |
| Anthropic bots | Supports the non-standard extension; the official example targets ClaudeBot. | Use the intended token. Do not infer an exact scheduling algorithm beyond the provider’s promise. |
| Yandex | Ignored since February 22, 2018. | Use the site crawl-rate setting in Yandex Webmaster instead. |
Named groups need their own restrictions
Bing’s current help and Crawl Control documentation describe support and precedence. Its older window explanation is useful for the semantics: one URL per time window is not necessarily an exact minimum gap between every pair of requests.
Anthropic’s current bot documentation gives a ClaudeBot delay example. Yandex’s notice explicitly retires the directive. Google lists Crawl-delay among unsupported fields. Those differences are why a copied all-bots delay can leave the traffic you care about unchanged.
In this authored configuration, repeat /drafts/ in each named group. Specific groups do not simply inherit the wildcard block. The hub’s group-selection example explains that boundary.
The seconds are example operating choices, not recommended settings for every site. This policy is not deployed on the author’s website.
User-agent: *
Disallow: /drafts/
User-agent: Bingbot
Disallow: /drafts/
Crawl-delay: 5
User-agent: ClaudeBot
Disallow: /drafts/
Crawl-delay: 1Reproduce pacing on a host you control
Download the local-host pacing example, extract it, and run node run.mjs with Node 22 or later. It binds only to 127.0.0.1 on an automatically assigned port. It starts a temporary policy endpoint, reads its Crawl-delay: 1, makes three paced page requests and three unpaced requests, prints the observed arrival gaps, then closes the server.
The synthetic clients use the same DelayFixtureBot identity and fetch the same pages, but only one waits for the configured delay after each response, so the unthrottled loopback server can receive the other client’s requests close together even while its robots file asks for spacing. The comparison isolates the client’s decision to wait.
The loopback example makes the client’s choice visible: the server asks for spacing and accepts both modes. Its strict spacing differs from Bing’s window algorithm, so it demonstrates the mechanism without simulating a provider.
No packages, external requests or persistent process. The printed milliseconds are measured on your machine.
unzip robots-crawl-delay-example.zip -d robots-crawl-delay-example
cd robots-crawl-delay-example
node run.mjsThe server owns the hard ceiling
For a production site, verify the caller through provider methods, then compare request start times, status codes, concurrency, and origin work. A request with a synthetic User-Agent tests your routing, not a provider’s compliance.
If the origin needs a hard ceiling, enforce rate, concurrency, quota and cache rules there or at the acting edge. Keep the policy endpoint available so cooperative clients can retrieve its preferences. A robots-file 403 or a permanently broken endpoint is not a reliable substitute for those controls.
For Google, crawl-rate guidance covers temporary 500, 503, or 429 responses, optional Retry-After with 503/429, and Search consequences when errors persist. If traffic changes after an outage, check cached policy behavior too. I would use Crawl-delay for cooperation and enforce the capacity ceiling where the requests arrive.
Path permission decides whether a fetch is allowed. Crawl-frontier scheduling and durable retries handle its timing and failed attempts.
Sources
- Google: supported robots.txt fieldsVerified Oct 26, 2026
Crawl-delay is unsupported.
- Bing: report an issue with BingbotVerified Oct 26, 2026
Crawl-delay support and time-window interpretation.
- Bing: Crawl ControlVerified Oct 26, 2026
Robots delay takes precedence over the console setting.
- Bing: window-based pacing explanationVerified Oct 26, 2026
Historical 2012 explanation; current support checked in the help pages.
- Anthropic: bots and site-owner controlsVerified Oct 26, 2026
Current ClaudeBot delay example and support for the extension.
- Yandex: Crawl-delay retirement noticeVerified Oct 26, 2026
Directive ignored since February 22, 2018.
- Google: reducing crawl rateVerified Oct 26, 2026
Capacity responses, Retry-After and prolonged-error consequences.
- RFC 9309: other recordsVerified Oct 26, 2026
Extensions can be interpreted without becoming standard REP rules.
Supporting material
- Owned local-host pacing example ↗Finite Node 22+ loopback server/client demonstration. No network targets, packages or provider impersonation.
- Pacing example source ↗Readable implementation of the temporary server and two synthetic client modes.