How to verify AI crawlers in your server logs

The first address may belong to your proxy

Behind a CDN or load balancer, the socket peer may be infrastructure the operator owns. Checking that address against a crawler feed can reject a genuine request before the classifier has reached the visitor’s address. Taking the first forwarded value is no general cure: a client or an earlier intermediary may have supplied it.

RFC 7239 treats Forwarded as information that can be modified by the client and by proxies. Trust in the proxy directly connected to the server does not automatically extend to the chain before it. The operator needs the platform’s documented client-address behavior and an explicit set of trusted hops; the classifier should record which policy selected the address. Header integrity depends on who can set or overwrite it; changing its name leaves that trust question open. Forwarding integrity.

Keep the socket peer, available raw forwarding metadata, request path and request identifier alongside the normalized address, subject to the operator’s retention policy. Then a proxy-configuration correction can create a new derivation from the same event. If ingestion retained only the chosen address and the daily count, the omitted chain cannot be reconstructed later.

The feed belongs to a product and a date

OpenAI publishes product-specific JSON ranges for OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot. Perplexity publishes a feed for PerplexityBot and another for Perplexity-User. The product mapping is therefore part of the evidence needed to classify the claimed identity. Anthropic supplies bot-network information through its crawler guidance. A match to some company-associated address is weaker than a match to the documented identity that the request claims.

Use an IP/CIDR parser that handles IPv4 and IPv6. Preserve match, no match, malformed address, stale snapshot and unavailable feed as separate states. A no-match result means that the address was unverified against the selected evidence. Investigate normalization, feed age, and retrieval failure before interpreting the result as spoofing.

Google’s verification guidance distinguishes common crawlers, special crawlers and user-triggered fetchers. Select the documented IP source or DNS method for the relevant category. The DNS method pairs a reverse lookup with a forward lookup confirming the requesting address; a familiar-looking reverse name alone is incomplete. Choose the product/category while classifying the request, before matching its network identity. Verification categories and methods.

Keep the claimed token and the matching network evidence as separate fields. If sources overlap, preserve the matches and the rule used to interpret them rather than letting whichever feed was loaded first define the product. This hypothetical overlap makes the logic explicit without asserting that the current feeds overlap. A published range supplies network evidence; purpose remains tied to the documented workflow and the request’s claim.

Attribution and delivery can disagree

Suppose the trusted address matches OAI-SearchBot’s saved range file and robots.txt permits the article, but the edge returns 403. That is an attributed request and an access failure. The repair belongs in the rule that denied the request, followed by a check of the delivered article.

A 200 challenge can withhold the document while a CDN cache hit can deliver it without an origin request, so join edge and origin records by request ID where possible, count the joined event once, and inspect the content or challenge classification before interpreting origin silence. The two cases require opposite delivery judgments.

For 429s, inspect the rate and throttling controls as well as the robots file. OpenAI’s advertiser guidance points operators to those responses and bot-mitigation events when crawler access fails. A working automatic search route also says little about a user-triggered fetcher, whose provider-specific robots behavior may differ.

A historical claim needs historical evidence

Today’s feed can supply a present-day match for last month’s address. Historical membership still needs a snapshot from that period. Keep the missing-history state in the row; a new pipeline can’t recover a feed nobody saved.

A revised normalization or matching rule should create another derived record. Keep the event fixed, attach the new trust policy, feed and parser version, then explain the rows that changed. This distinguishes a traffic change from a classification change. A reported match rate is also distinct from classifier accuracy: measuring false matches and missed matches would require independent reference labels, not agreement with the same feed used to produce the result.

The proportionate method depends on the decision. For one suspected WAF denial, a documented manual verification and the returned response may answer the question. A recurring historical traffic report needs reproducible snapshots, joins and derivations. Requiring the latter for every incident wastes effort; using the former to imply a reliable longitudinal count overstates the evidence.

The result should say what happened at the measured layer: this request claimed this product, its selected address matched this dated source, and this response delivered the article or withheld it. Counts with unknown attribution and unknown delivery should remain visible alongside successful matches. Then the report can guide a specific access repair without pretending to measure search inclusion, answer citation or training use.

A public result needs the method, not visitor addresses

Attach the exact feed bytes, retrieval time, and digest to the classification run. OpenAI’s documentation maps products to their feeds and purposes, so the source snapshot should travel with the decision it supports.

For DNS verification, Google requires reverse and forward checks. Verify the allowed hostname ending and return to the original address under the correct crawler category. A name merely containing “google” can’t substitute for those checks.

Publish aggregate observations and reproducible method evidence, keeping unnecessary visitor addresses private. If the data contains only declared agents, I would call it unverified agent activity. A smaller, accurately named count is more defensible than an inflated verified total.

The dated 90-day crawler observations mark crawler identities as unverified. The llms.txt request count reports fetching, leaving reading and citation unmeasured.