Do AI crawlers respect robots.txt? Controlled tests find differences

What the controlled studies add

A 2025 preprint, Scrapers selectively respect robots.txt directives, studies institutional logs over 40 days and 130 self-declared bots under changing rules. It reports selective compliance and weaker compliance under stricter directives. Its setting and self-declared identities limit how directly those results transfer to a different website or current product.

A July 2026 preprint, Do Generative AI Assistants Respect robots.txt?, tests ten assistants across 200 controlled trials. It combines server observations with secret page codes and varies universal and agent-specific policies. The authors report substantial differences and show that page access can diverge from visible answer correctness.

The studies supply experimental structure for particular clients, configurations, and dates. A later product or another retrieval route needs its own test. sulayman-bowles.dev’s passive User-Agent counts describe a different measurement.

A policy promise, a request, and a violation need different evidence

RFC 9309 defines cooperative crawling rules, including policy caching and error handling. Enforced denial belongs at the edge or application, under the protocol’s access distinction.

Provider purpose matters too. OpenAI distinguishes automatic crawling from user-triggered ChatGPT-User requests, to which robots rules may not apply. A request associated with an assistant should not automatically be graded as a training crawler. Use the current agent map before defining the test.

The provider states a policy. The site observes a request. A violation finding joins a verified identity to a prohibited retrieval under the policy applicable at that time. Keep that chain visible so the owner can locate the next test.

What my logs show, and what they leave unknown

The 90-day snapshot contains 26,083 self-identified-bot events and 4,265 suspected-automation events. All identities are unverified. The observer records selected request arrivals through a best-effort pipeline; it does not store returned HTTP status or content.

It also lacks a versioned timeline of the robots bytes served at each request time. Comparing a July request with today’s file would substitute a later policy for the one the client could have read. A request shortly after a policy change also requires attention to caching and propagation.

I report compliance as unmeasured. The existing events can identify cases to verify while leaving both compliance and violation claims undecided.

What would make the next test persuasive?

Use an authorized, harmless test surface and archive each policy version with its effective time. Keep page content, public discoverability and client configuration stable enough to interpret the policy change. Test allowed and disallowed conditions with repeated runs instead of judging one screenshot.

Record the requested path and method, trusted identity evidence where available, response status, and a content marker, then compare that served document with the visible answer while keeping robots requests and cache intervals alongside the policy version so a correct answer can’t conceal a prohibited retrieval or a failed fetch. Either stage can succeed while the other fails.

Predefine the classification: compliant observation, prohibited retrieval with adequate evidence, attempted request denied by the edge, identity unresolved, policy unresolved or no observable attempt. Preserve uncertainty in the denominator. Excluding every ambiguous case silently would make a rate look stronger than the measurement.

How the owner can act before that experiment exists

Keep robots.txt aligned with the publishing decision, and use real authorization for private material. If the immediate issue is load or an unwanted request, inspect the acting edge rule and delivery evidence. You can repair that incident without first proving a platform-wide behavioral claim.

The access guide locates policy and delivery failures, while the verification guide tests identity. I would use them to resolve a specific request first. A platform-wide compliance claim should wait for a test designed to measure it.