Robots.txt Allow and Disallow: rules and examples

The selected group determines Allow and Disallow

A crawler directive needs a URL and a matching rule before it has an observable result. In this example, every test URL is on https://example.com. ExampleCrawler uses the wildcard group; Googlebot has its own group. Google’s group-selection rules explain why the second group does not inherit restrictions from the first.

Within the selected group, the most specific matching path wins. An equally specific Allow takes precedence over Disallow. Matching is case-sensitive, and a final $ anchors a rule to the end of the path and query. RFC 9309’s matching rules and Google’s wildcard examples describe those decisions.

User-agent: *
Disallow: /drafts/
Allow: /drafts/public-guide$
Disallow: /search?

User-agent: Googlebot
Disallow: /staging/
Allow: /staging/release-notes$

Eight paths, two crawler groups

The eight-case download places each expected decision beside its path. Run the example policy through a parser to compare them. The result tests rule matching; requests and later uses need their own observations.

• Allow does not override every Disallow: specificity decides between matching rules.
• Disallow: / asks a cooperative crawler to avoid the whole site. An empty Disallow value imposes no path restriction.
• A missing matching rule permits the fetch. Keep a separate server-side permission check for private resources.
• A copied User-Agent does not prove crawler identity; this table evaluates a claimed product token only.

The AI-crawler policy examples apply these rules to named products. The crawler-compliance review compares documented intent with observed behavior.

The desired outcome chooses the control

A public faceted catalogue can rely on cooperative crawl preferences. An account document must stay private when a client lies about its identity or ignores the policy. Putting both jobs in robots.txt leaves the harder one unenforced.

For public material, there is another consequential distinction. Google can know a URL exists without fetching the page, so a robots disallow can leave the address in Search. To remove a public page through noindex, Google has to retrieve the directive. Blocking that retrieval first can prevent the indexer from seeing the instruction you meant it to follow. Google’s robots limitations.

The crawler directives decision table connects the result you need to the directives in a concrete response.

No robots.txt is not the same as noindex

A missing robots file does not mean “do not crawl.” Under RFC 9309’s access-result rules, an unavailable file such as a 404 may permit access; a server or network failure is a different state. Google documents its own robots.txt status handling, including a special treatment for 429.

To exclude an otherwise public HTML page from Google Search, put noindex in the page’s head or its HTTP response, and let Google retrieve it. Google does not support a noindex line in robots.txt. Blocking the URL before Google can fetch that instruction can leave the URL discoverable in Search. Google’s noindex guide gives both supported forms.

• Fetch the deployed robots response, including redirects and status, before blaming a rule in your local file.
• Then inspect the page response and rendered metadata separately. The raw HTML and rendered DOM evidence guide shows what each observation can establish.
• Use the server-log verification method to investigate the caller, rather than treating its User-Agent as proof.

<!-- HTML head -->
<meta name="robots" content="noindex">

# Alternative HTTP response header
X-Robots-Tag: noindex

A cached copy can defeat the permission check

Consider an account document loaded by an authorized user. The application checks permission and returns the file. A shared cache then stores that response under a key that omits the user’s identity. When an anonymous client requests the same address, the cache serves the private document without asking the application again.

I would test the anonymous response before and after an authorized visit across public hosts and alternate routes, inspecting the returned body because a cache that serves an earlier user’s file can defeat correct authorization code while the response status still looks like a successful request. The application passed; delivery leaked. A 200 can carry a login page or the private file.

RFC 9111 gives cache directives different jobs. Unqualified private excludes shared-cache storage while permitting private caches under their applicable rules. Unqualified no-cache permits storage but requires successful validation before reuse. No-store prohibits cache storage. None of these decides who may receive the original response.

The cache and permission policies must preserve the same access decision through delivery. Inspect public headers and bodies, then test an anonymous client and another signed-in principal after an authorized visit. A protected application response won’t cover a file route that bypasses it or an edge rule that replaces it.

Alternate routes, methods, and file variants need the same protection. A cache can change which component returns the body, so authorization has to survive the whole delivery path. Robots policy contributes no permission check to that chain.

curl -fsS -D robots.headers https://example.com/robots.txt -o robots.body

curl -sS -o /dev/null -w "%{http_code}\n" \
  -A "ExampleCrawler/1.0" https://example.com/private/account

curl -sS -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0" https://example.com/private/account

Cooperation still needs a real parser

Cooperation fits a public catalogue’s crawl preference. Its parser still needs correct product-token matching, group selection, percent-encoding, and longest-path rules. I would use that lighter control for public crawling and require enforced authorization for confidentiality.

The retrieval result matters too. RFC 9309 permits access when the file is unavailable, such as a 404, and requires assumed complete disallow when server or network errors make it unreachable, subject to its caching and prolonged-failure provisions. A local “fail closed on every error” setting is an operator override. Record it as one. Protocol access results.

Save the returned file with its effective URL, redirect chain, status and retrieval time. Then record the parser version, selected group and matching rule. Together they let a reviewer explain why production made a different decision from the copy under review. Scheme, host and port define the serving service. Caches belong in this account too.

The robots.txt error and cache timeline follows a failed policy retrieval through Google’s documented last-good-copy behavior.

A familiar name is a claim

Any client can send a Googlebot or GPTBot User-Agent. That string can help select a robots group and route a log record, but granting privileged access from the string alone lets the requester name its own permissions.

Provider-published ranges and verification methods can improve attribution for a public workflow. Save the source version, retrieval time and trusted client address, with explicit stale or missing-evidence states. Even a well-attributed crawler has acquired no right to another user’s account document. Attribution can inform an operational rule without becoming the resource’s authorization policy.

Rate limits, concurrency limits, quotas, and caching decide how much work the origin accepts. An allow rule can support discovery while the server limits expensive requests, giving public crawl policy and capacity enforcement their own jobs.

The release has two observable results

A release needs two independent results: the public robots file selects the intended crawl rules, and protected bodies stay unavailable to anonymous and wrong-user requests after cache warm-up. Keep the returned file and parser decision for the first, then response evidence for the second.

Logs can explain who reached the measured service and which response followed. For public pages, cooperative rules can be proportionate. For private documents, I would require the stronger result: an uncooperative client still can’t obtain the body from any component that serves it.