Is Cloudflare or your host blocking AI crawlers? The denial shows why

Where does the request stop?

Follow the request from public policy to edge enforcement and then to the application. A robots disallow asks a compliant crawler to refrain from requesting a path. A WAF can stop the HTTP request. An application can return a login form even after the edge passes it.

A publisher can intentionally allow search while declining training, or open help pages while requiring sessions for account data. Write down that result before making every dashboard indicator green.

Does the edge serve the file you edited?

Fetch /robots.txt from the exact public hostname. Save its bytes, response headers and check time. Compare it with the source version before assuming a recent commit changed the live policy.

Cloudflare’s current robots setting can prepend managed directives to an existing origin file. Its October 8, 2026 documentation describes Bot Preference Sync using Search, Agent and Training policies. Inspect the combined response, especially when a managed group and an origin group name the same crawler. Managed robots.txt behavior.

Current Cloudflare guidance also offers behavior-specific Search, Agent and Training policies. Check the active values on your zone instead of following an old screenshot of a single “Block AI bots” toggle. AI bot policy controls. You may want the training block to remain in place while repairing search access.

What did the denied request receive?

Save the GET body as well as headers. It distinguishes an article from an error or challenge that shares the same status. The policy probe checks robots.txt; the example below saves the page itself.

• For a denial, find the matching security event using its time, host, path and request ID.
• For a challenge, inspect headers and body; Cloudflare documents cf-mitigated: challenge.
• For 429 responses, inspect the active throttle and request volume.
• For 200 responses, confirm that response.html contains the expected page text.

curl --connect-timeout 10 --max-time 30 --silent --show-error --location \
  --dump-header response.headers --output response.html \
  --write-out 'final_url=%{url_effective} status=%{http_code}\n' \
  https://example.com/public-article/

How would a scoped fix work?

Consider a public service page whose robots policy permits OAI-SearchBot. A later request claims that agent, matches the relevant dated provider network evidence, and receives 403 at the edge. The security event points to a country restriction that runs before the crawler rule. This is the illustrative incident; each field would need evidence in a real case.

If a country rule causes the denial, decide whether the verified product should receive an exception on the intended hostname and public path, then change that expression while saving the previous rule and rollback route so the next response tests the precise repair. An Allow line in robots.txt leaves that HTTP rule untouched.

When the exception depends on identity, pair the claimed agent with maintained provider evidence or a documented verified-bot mechanism. OpenAI publishes product-specific sources, as does Perplexity. Admit the intended product, not any request that copies its name.

Cloudflare’s Skip action can target selected security products but can’t bypass Bot Fight Mode. Check the product before designing the exception. A blanket bypass for requests containing “bot” would expand a one-page repair into a much wider permission.

What evidence closes the incident?

Repeat the diagnostic and inspect the delivered article. Test an excluded or protected path too. The cf-mitigated header helps identify a Challenge Page; the response body still decides whether the requested content arrived.

A request made with curl -A GPTBot still originates from your machine. It can reveal a string-based rule, but it cannot reproduce OpenAI’s source address, fingerprints or request behavior. A provider access claim needs a later provider-attributed event, classified with the server-log method.

If the edge serves a cached article, the origin may have no matching hit. Preserve edge evidence and join logs where possible. If no attributable provider request arrives during the observation window, leave that final check pending and record the working local response.

I would close the incident with the old denial, the rule changed, the new document response, and the provider-check result. The repaired public path should work for its intended caller while the owner’s training and private-content decisions still hold.