Crawler evidence
90 days of requests: 49,152 events, no verified crawler identities
The log records 49,152 events in three groups: claimed bots, suspected automation, and ordinary browsers. These labels leave the crawler’s identity and delivery of the requested page unverified.
sulayman-bowles.dev accepted 49,152 observer events in 90 days. Of those, 30,348 fell into bot or automation classes, about 61.7% of the total. I would use that share to choose logs for investigation, because every identity is unverified and the observer doesn’t record whether the requested page arrived.
The classifications are 26,083 self-identified-bot events, 4,265 suspected-automation events, and 18,804 ordinary-browser events. A ClaudeBot label describes the request’s claim. It leaves Anthropic’s identity and later use unconfirmed.
Method & limits
Production observations for sulayman-bowles.dev only, July 23, 2026 at 20:48:00.901287 UTC inclusive to October 21 at the same time exclusive: exactly 90 elapsed days. Queried October 21 at 20:50:26 UTC. The observer is selective and best effort. Every event is unverified; no response status, delivered content, indexing, training or citation outcome is measured. Boundary calendar days are partial; this is not a continuous-delivery audit.
Three classes share one denominator
The three classes sum to 49,152 events. Ordinary-browser is a classifier label too: it doesn’t certify a person, just as unmatched automation has no verified company behind it.
| Observed class | Events | Meaning |
|---|---|---|
| Self-identified bot | 26,083 | User-agent pattern looked like a bot |
| Suspected automation | 4,265 | Automation indication without a named provider claim |
| Ordinary browser | 18,804 | Ordinary-browser classifier result |
| All classes | 49,152 | Accepted events; not unique clients |
Which familiar names appeared?
These selected registry labels describe claims inferred from User-Agent patterns. The stored registry was version 2026-07-11 and the classifier was 2026-07-11.1 throughout the snapshot. A label can be copied by another client, and a later registry could classify an old string differently.
Another 15,923 self-identified-bot events had no named agent match. Leaving those unmatched is more informative than assigning them to a company from a weak hint. The full grouped table is in the snapshot bundle.
| Claimed agent | Recorded events |
|---|---|
| ClaudeBot | 1,563 |
| ChatGPT-User | 1,177 |
| PerplexityBot | 997 |
| OAI-SearchBot | 716 |
| GPTBot | 366 |
| Perplexity-User | 317 |
| Claude-User | 97 |
| Claude-SearchBot | 5 |
Ninety elapsed days leave capture gaps possible
The fixed interval intersects 91 UTC dates because it begins and ends partway through a day. At least one event appears on every intersecting date. That checks for whole missing date bins; it cannot rule out shorter outages, dropped deliveries or requests excluded before ingestion.
The middleware selects public GET and HEAD documents plus certain machine-readable resources, excluding APIs, health checks, most static assets, and other methods so that the saved rows describe its chosen observation surface and successful best-effort deliveries instead of the full traffic reaching the host. Logging fails open to preserve the visitor’s request.
The window can include the owner’s browsing, synthetic checks and diagnostic clients. No complete organic-versus-test label was available in this extraction. Dividing these counts by days produces an accepted-event average, not a provider’s true crawl rate or a measure of audience size.
What decisions can these records support?
They identify names and routes worth investigating. A surge in a claimed agent can lead to an edge-log check; an unmatched class can lead to classifier review. Before changing a provider-specific rule, follow the identity-verification method and inspect what the request actually received.
A robots-compliance test needs the policy delivered at the time, applicable token and path, verified client identity, and treatment of caching and user-triggered fetches. The compliance guide describes the joins this observer lacks.
The llms.txt study follows one path’s requests more closely. Reading, training, and citations still require evidence after the request stage.
Frozen aggregates make the arithmetic reproducible
The download includes daily class totals, claimed-agent aggregates, llms path/method counts, version counts, the aggregate SQL and a method note. Sum the daily table by observed class to reproduce the headline. Sum the claimed-agent groups to reconcile the 30,348 non-browser-classified events.
The public projection omits raw events, full user agents, IP-related fields, geography, query names, referrers, and deployment identifiers. Live ingestion and retention can change a later query; use the frozen .dev files to reproduce this window. I would trust them for the accepted-event arithmetic and use independent request evidence before acting on a provider name.
Sources
- Frozen .dev observer aggregates, SQL and methodSulayman BowlesVerified Oct 21, 2026
- Observer window, privacy projection and measurement limitsSulayman BowlesVerified Oct 21, 2026