Crawlers, search & the evidence behind them.

An AI crawler guide: separate search from training, check robots.txt and firewall delivery, verify bot identity, and use Google’s current AI controls.

October 21, 2026 study of 67 Austin home-service brands: six bot-policy checks, ordinary HTTP observations, five unknown policies, and data to download.

Check AI crawler access using the live robots.txt file, saved GET responses, edge events, and attributed requests. See what a crawler checker can verify.

Diagnose AI crawler blocks in Cloudflare, a CDN or your host using delivered robots.txt, saved responses, security events and scoped access fixes.

Compare crawl-delay support for Googlebot, Bingbot, Anthropic, and Yandex. Run an owned local-host example to inspect paced and unpaced requests.

Explore a frozen 90-day .dev request-observer snapshot, with daily counts, claimed crawler names, downloadable aggregates and explicit coverage limits.

Compare controlled robots.txt research with sulayman-bowles.dev’s unverified request observations; see the evidence needed before calling a request a violation.

Does anyone fetch llms.txt? sulayman-bowles.dev recorded 249 /llms.txt requests in a fixed 90-day window. Compare GET and HEAD, claimed agents, downloadable counts, and what they cannot prove.

Compare Google-Extended, Search Console’s Search generative AI control, Googlebot and snippet directives, including property inheritance and verification.

Diagnose robots.txt 404, 403, 429 and server errors. Compare RFC 9309 with Google’s 12-hour pause, last-good policy reuse and documented cache behavior.

Four complete robots.txt examples for AI crawlers, with RFC grouping rules, search and training distinctions, user-fetch limitations and validation steps.

Use robots.txt Allow and Disallow with worked examples. Trace eight cases, compare named groups, and choose noindex or authorization for other goals.

Robots.txt wildcard matching, path prefixes, Allow/Disallow precedence, and the $ end marker. Run 18 named cases with expected results.

Decide whether to block GPTBot, keep OAI-SearchBot available for ChatGPT Search, and verify the robots.txt change without confusing access with citations.

Design a crawler frontier with URL identity, origin scheduling, bounded retries, and crawl-trap controls that leave omissions and failures auditable.

Verify GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and other AI crawlers in logs using IP-range checks, User-Agent parsing, and status codes.

Align your canonical website, résumé, and public profiles using a reconciliation log, sameAs criteria, and dated historical records that remain accurate.

Generate the title, canonical, author and structured data from the document, preserving the precision of its dates.

Choose equivalent content, then test redirect and canonical edges for conflicts, chains, and loops. Covers metadata, missing tags, and duplicate titles.

Diagnose Crawled – currently not indexed: compare indexed and live evidence, inspect HTML and canonicals, and choose a repair that adds value.

Choose by outcome: robots.txt for fetching, meta or X-Robots-Tag for indexing, canonical for duplicates, and server controls for private access.

Interpret “alternate page with proper canonical” and “duplicate without user-selected canonical.” Inspect both URLs and preserve distinct content.

Fix “indexed, though blocked by robots.txt” by choosing inclusion or removal. Compare blocked and readable noindex responses, then verify the repair.

Map internal links with source context, canonical destinations, and entry paths. Diagnose orphans and validate repairs without assuming ranking gains.

Resolve a canonical noindex conflict by choosing exclusion, consolidation, or independent indexing. Compare final HTML, HTTP headers, and crawl access.

Two different articles can share a template title. I’d compare their content before changing a canonical; title fixes needn’t consolidate either document.

A missing tag is a fact about a response. I’d choose the preferred accessible document, then express that choice in its head, links, and sitemap.

Fix multiple canonical tags with three-layer checks of raw HTML, HTTP Link headers, and rendered DOM. Remove competing generators and verify the repair.

Fix an OG URL/canonical mismatch with paired HTML examples, tracking and wrong-host cases, intentional pagination differences, and a capture checklist.

Fix canonical pagination with a nine-item example. Compare self-canonical category pages with a true view-all page, and test crawlable item discovery.

Check React and Vite public routes before release: static HTML, HTTP status, canonical URLs, crawlable links, assets, and hydration, with a test table.

A two-page capture shows which text arrives in HTML and which part still needs a browser observation.

Plan SEO migrations with a URL manifest, redirect-graph checks, and page contracts. Test the public edge and distinguish site defects from index changes.

Sulayman Bowles built Atlas, a local Python and SQLite website audit console connecting crawls, findings, source evidence and exports.