Crawler evidence
AI crawler access in Austin: 13 allowed brand pages still had delivery gaps
Most tested homepage paths were allowed by the saved robots rules. Policy and a delivered page still produced different results.
Thirteen of 61 Austin home-service brand records allowed OAI-SearchBot in robots.txt but lacked a complete 200 homepage response to the research client. That’s about one in five. I see the delivery gap as the main finding: a policy-only check would have missed it.
Across all 67 brands, 61 allowed that token at the tested path, one disallowed it, and five remained unknown. Only 48 returned a complete HTTP 200 homepage body without challenge indicators. These were ordinary research-client requests, so the gap calls for delivery diagnosis before any claim that OpenAI itself was blocked.
Method & limits
Observed October 21, 2026, 20:08:13–20:08:53 UTC. One analytic record per brand in the 67-brand July ownership panel: 66 official URLs on 65 requested origins, plus one unresolved URL. Requests used an ordinary research client; provider-network access, JavaScript rendering, and AI citations were not tested. Ownership remains dated July 26, 2026. Unknown and incomplete observations stay in the denominator.
What the six policy checks found
The table evaluates the saved robots rules for each brand’s final homepage path, or its requested path when no redirect was observed. “Allowed” means no matching restriction in a usable policy under the documented parser. It does not mean that a provider visited, indexed or cited the site.
The training-oriented tokens and search-oriented tokens can receive different rules. The crawler policy guide explains those product distinctions. The unit is a path; a domain can have other pages with different responses.
Redirects change the authority being evaluated. A-Tex’s saved homepage redirect ended at a Canva design editor/share URL; the OAI-SearchBot and Googlebot disallow in that row belongs to Canva, not to atexpm.com. Its requested-origin robots response was 404 and remains unknown in this study. The table therefore describes final targets, and must not be read as a list of brands intentionally blocking crawlers.
| Product token | Allowed | Disallowed | Unknown |
|---|---|---|---|
| GPTBot | 60 | 2 | 5 |
| OAI-SearchBot | 61 | 1 | 5 |
| ClaudeBot | 60 | 2 | 5 |
| Claude-SearchBot | 61 | 1 | 5 |
| PerplexityBot | 62 | 0 | 5 |
| Googlebot | 61 | 1 | 5 |
A successful policy check can precede an incomplete fetch
The homepage requests produced 51 HTTP 200 statuses, ten 403s, three 404s, and three records without a final status. Only 48 of the 51 HTTP 200 bodies were complete under the collection cap. Nine records across all statuses carried challenge indicators. Those results describe the research client’s delivery.
The three missing final statuses include the unresolved official URL and transport gaps. The observations give each its status, completeness, error, redirect, and challenge fields so collection gaps stay visible.
A site owner who sees an allowed policy and a denied or incomplete research response has a next diagnostic step: inspect the matching security or delivery event. Changing the robots file again may address the wrong layer. The access-checking guide separates the checks.
The cohort comes from a dated ownership census
The sampling frame is the July 26 Austin home-service census. Its complete intake has 75 records; 67 are published_active. Historical, excluded and unpublished-candidate rows stay outside this study’s brand denominator.
Existing source evidence grounded official URLs for 66 brands. Two brands shared a requested origin, leaving 65 requested origins; some records use national or platform-hosted sites. One brand had no supported official URL and remains in the table as unresolved. Replacing it with a search guess would make the roster look more complete without making the observation more trustworthy.
The preserved owner classifications are historical. They are available for descriptive grouping, but this run does not reverify current ownership or show that an owner caused a policy decision. Shared franchise infrastructure also prevents treating every brand row as an independent server system.
A bounded check with saved policies
The collector used four workers, a minimum 1.5-second spacing between requests to an origin, at most five redirects, finite connection/request timeouts and one transient retry per hop. The run made 161 request attempts including redirects. Homepage bodies were capped at 1 MiB; robots bodies at 512 KiB. No bot-labelled simulation was used.
For a usable saved policy, the evaluator matches named product tokens without case sensitivity, merges repeated matching groups, falls back to the wildcard group and applies path rules with allow winning a tie. The code, parser tests, roster and saved parseable policy bodies are in the research bundle. The implementation is an inspectable protocol subset, not a provider-conformance certificate.
Unreadable, incomplete, or failed policies are unknown under this conservative study classification. A crawler can have another error fallback. The method explains that choice and reconciles 468 requested/final-target policy rows to the 67 analytic brand records.
The delivery gap identifies the next check
For one of your own public pages, save the policy, the response and the useful content separately. If policy disallows the page, resolve that choice first. If policy allows it but the edge returns a challenge, investigate that response. If the document is delivered, a real provider visit or answer citation still needs its own observation.
The dated observations can be repeated against the same brand identifiers. Their scope is public policy and delivery, leaving contractor quality, market share, licensing, intent, and llms.txt effects outside the measurement.
Keep brand, platform, and historical owner fields identifiable when joining a later check, as the dataset guide describes. I would investigate the 13 delivery gaps before widening crawler permissions: the saved policy already allowed those paths.
Sources
- Public homepage observations — October 21, 2026Verified Oct 21, 2026
- Study bundle: roster, policies, parser, tests and configurationVerified Oct 21, 2026
- Study method and classification boundariesVerified Oct 21, 2026
- Original Austin ownership relational dataset — July 26, 2026Verified Jul 26, 2026
- RFC 9309 — Robots Exclusion ProtocolVerified Oct 21, 2026
- OpenAI — crawler product purposes and policy choicesVerified Oct 21, 2026
Supporting material
- 67-brand observations ↗One analytic row per brand with policy, status, completeness and unknown fields.
- Reproducible study bundle ↗Roster, policy rows, saved parseable robots policies, collector, parser and tests.
- Method note ↗Denominators, collection settings and exact limits on the claims.