How to check whether your site blocks ChatGPT, Claude, or Perplexity

Which kind of access are you testing?

A checker that asks whether “ChatGPT is allowed” has already compressed several decisions. Search discovery, training collection and a person asking an assistant to open a URL can follow different policies. Use the product token in the report.

OpenAI says ChatGPT-User rules may not apply, Anthropic says its user fetcher honors robots, and Perplexity says its user fetcher generally ignores them. Select the exact product token before interpreting the policy.

What does the public host serve?

Use the final public URL, including its scheme and hostname. The policy on https://example.com does not cover a separate www or help hostname just because the pages share a logo. Google’s robots specification describes the host, protocol and port scope.

Save GET headers and bodies for the policy and page, including error responses and final redirect destinations. Run the commands from an empty working folder and replace both example URLs.

curl --connect-timeout 10 --max-time 30 --silent --show-error --location \
  --dump-header robots.headers --output robots.body \
  --write-out 'final_url=%{url_effective} status=%{http_code}\n' \
  https://example.com/robots.txt

curl --silent --show-error --location \
  --dump-header page.headers --output page.html \
  --write-out 'final_url=%{url_effective} status=%{http_code}\n' \
  https://example.com/services/

A failed fetch may come from Cloudflare or host controls. Check the delivered response before changing the crawler policy.

Does the matching group allow the intended path?

Feed the saved policy into a parser that accepts a product token and full URL. Keep the parser name and version. Google publishes a robots.txt parser and matcher with a single-URL command; its documented build instructions provide one reproducible option, although another provider’s implementation may differ.

Test every selected search token against both paths, then test the training token against a public article if you decline training collection. RFC 9309’s group rules make * a fallback and combine repeated groups for one product token.

Did the request receive the article?

Inspect page.html for the main heading and a distinctive sentence from the intended article, because a login form, challenge, empty application shell, or error page can all make a technically successful HTTP response fail the publishing task while leaving the status at 200. Save that body with the classification.

If a browser displays the missing text after JavaScript runs, record that as a browser observation. It is an additional representation to inspect. The raw HTML and rendered DOM method explains how to capture a browser’s own document response without assuming a separate curl request received the same bytes.

For Cloudflare challenges, cf-mitigated: challenge identifies a Challenge Page response. Cloudflare’s header reference gives the documented signal. Preserve any request ID and time so you can find the corresponding security event.

What evidence comes from the provider itself?

Now inspect edge logs for the chosen path and time window. A request header naming a provider is a lead. Use the trusted client address and the provider’s current verification method before attributing it; OpenAI and Perplexity link product-specific IP feeds from their documentation. Preserve the evidence date instead of silently classifying historical requests against a new list.

Changing curl’s User-Agent tests a string-based routing difference on your network. Provider visits need their source network and request evidence, including any relevant TLS fingerprint, request sequence, or credentials. Keep the probe in a synthetic-check class.

The server-log verification guide covers proxy addresses and missing attribution evidence. If there is no attributable request in the window, mark provider delivery unverified. Do not turn silence into either a pass or a block.

What should the final access report say?

The report should locate the next repair. “Policy disallowed” points to robots.txt. “Policy allowed, edge denied” points to security rules. “Document delivered, no attributed provider request” calls for observation before another policy change.

Keep the exact URL, check time, token, saved policy and response, and attribution status together. The provider map supplies the choices. I would repair the failed control first, then measure citations under their own recorded conditions.