Researchai crawlers
Back to writing

Crawler evidence

Does anyone fetch llms.txt? sulayman-bowles.dev recorded 249 requests

249 recorded requests to one path; delivered content, reading, and downstream benefit remain unmeasured.

Monochrome generative point study “Tidal Bell,” a mathematical form rendered in black and white.

The observer recorded 249 requests to /llms.txt in 90 days: 244 GET and five HEAD. I would keep a short published file accurate, but that count supplies no evidence of reading or benefit. The observer saves request arrivals, leaving returned status and content unmeasured.

A file with working canonical links can have a modest editorial purpose. Its count doesn’t justify replacing readable HTML, ordinary links, or search discovery with an llms.txt strategy.

Method & limits

The .dev-only snapshot uses the linked observer study’s exact 90-day interval ending October 21, 2026 at 20:48:00.901287 UTC. It records accepted arrivals, leaving response status, content, and authenticated identities unmeasured. Reading and causal effects on indexing, citations, or traffic remain untested.

What is in the 249-request count?

Count the exact path. /llms.txt contributes 249 records, while /llms-full.txt contributes another 27 outside the headline. A client can probe either name even when no document exists there.

The five HEAD requests have no document body. GET requests can ask for one, but this dataset stops before recording the returned status or content. Keep those methods identifiable when choosing a delivery check.

Exact /llms.txt path, fixed 90-day .dev window. Client identities are unverified.
Observed classGETHEADTotal
Self-identified bot1285133
Suspected automation46046
Ordinary browser70070
All classes2445249

Which names did clients claim?

The named labels include bingbot, Googlebot, GPTBot, OAI-SearchBot and ChatGPT-User. They come from User-Agent classification. They are not authenticated provider identities, and some records may be diagnostic or synthetic checks.

The small named totals identify a verification sample. Without authenticated client identities and session-to-answer linkage, they leave provider adoption and use of linked content unknown.

Selected named User-Agent classifications for GET /llms.txt. All are unverified.
Claimed nameGET observations
bingbot28
Googlebot14
GPTBot7
AhrefsBot6
OAI-SearchBot3
ChatGPT-User2
Chrome-Lighthouse2

Why a request is a weaker result than a reader

To turn a request into a reader or outcome, independently identify the client, confirm a usable response, verify content consumption, and join that use to an answer or referral, because a client can request a missing path or fetch a file it later discards while producing the same arrival event. This study stops at that first event.

Absence of an observed provider label would also be ambiguous. Retrieval could use a different identifier, a search index, a cache or an intermediary, and the best-effort observer could lose a delivery. The broader snapshot describes the selected routes and capture limits.

The window was not a randomized deployment experiment. Other content, links, publication and search conditions can change during ninety days. Comparing traffic before and after an llms.txt edit would need a design that addresses those changes; these counts do not isolate the file’s effect.

What I would maintain, and what I would measure next

I would keep a published file accurate, concise and connected to canonical public resources. Broken links and stale claims make it a poor guide for any client. That maintenance decision can stand on its small cost and editorial purpose without claiming a measured citation benefit.

For a stronger test, define the desired outcome first: delivered file, subsequently fetched linked document, answer citation or referral. Capture the fields needed to join those stages within the site’s privacy constraints, verify client identity where possible, and retain a comparison. The server-log guide is the starting point for the attribution step.

The result remains 249 recorded requests, including five HEAD requests. A stronger test would begin at delivery and follow a verified client beyond that request.

The two paths reconcile to 276 requests

The CSV preserves exact path, method, observation class, claimed label, verification state, count and first/last event times. Sum only /llms.txt for 249; include /llms-full.txt to obtain 276 across the two names. Blank claimed labels remain unmatched.

The snapshot bundle supplies the extraction SQL, daily denominator, and method without raw visitor records or private network identifiers. I would maintain the file for its small editorial job, then require a delivery-to-outcome comparison before spending on a larger llms.txt strategy.

Sources

  1. Frozen .dev observer aggregates, SQL and methodSulayman BowlesVerified Oct 21, 2026
  2. Observer window, privacy projection and measurement limitsSulayman BowlesVerified Oct 21, 2026
  3. Exact-path and method aggregate CSVSulayman BowlesVerified Oct 21, 2026

Read next