Crawler policy
Robots.txt wildcards: which path does the rule match?
The slash, the query string and the selected crawler group can change which rule wins. Read the decision beside the exact input.
A robots.txt block on /shop also blocks /shopping. A final $ ends the match, while * matches zero or more characters, including slashes. To test a rule, select the crawler’s group, then compare the exact path and query. The most specific matching rule wins, with Allow winning an equivalent tie. Matching is case-sensitive.
The 18-case fixture below tests the details that break policy generators: directory separators, empty wildcard matches, suffixes, query strings, and encoded characters. It extends the eight-case policy example.
Method & limits
Primary documentation checked October 26, 2026. The example.com responses and local fixture are authored teaching examples; they do not report a Google indexing result or observed provider compliance.
The path and query decide the match
A rule is not a shell glob over filenames or a regular expression over a full URL. It applies to the path and query of a URL on the service that supplied the policy. A fragment is not sent in the HTTP request. Choose the User-agent group before comparing paths.
A trailing slash is consequential: /drafts/ names a directory prefix, while /drafts also matches /draftsmanship. A final $ makes the endpoint exact. In /*.pdf$, the wildcard can include slashes, but the suffix must end the path-and-query string; report.pdf?download=1 does not end in .pdf.
RFC 9309’s special characters define * and $. Google’s path examples show their use with case and query strings.
| Rule in the fixture | Request path and query | Expected decision | Why |
|---|---|---|---|
| Disallow: /shop | /shopping | Blocked | A prefix does not imply a directory boundary. |
| Disallow: /drafts/ | /draftsmanship | Allowed | The slash is part of the required prefix. |
| Disallow: /assets/*.pdf$ | /assets/report.pdf?download=1 | Allowed | The query means .pdf is no longer at the end. |
| Disallow: /assets/*.pdf$ | /assets/.pdf | Blocked | * can match zero characters. |
| Allow: /drafts/public$ | /drafts/public?lang=en | Blocked | The exact exception misses; /drafts/ still blocks. |
| Allow and Disallow: /tie | /tie | Allowed | Equally specific Allow wins. |
The runner checks all 18 local decisions
Download the executable fixture bundle, extract it, and run node run.mjs with Node 22 or later. It reads only its adjacent robots.txt and cases.csv, prints each expected and actual decision, and exits unsuccessfully if a case differs. It makes no network requests and installs no packages.
The policy, expected results and runner are also available separately. Run from any directory: inputs resolve relative to the script, not your current working directory.
The runner models exact product-token groups, combined equal groups, fallback *, path matching, and normalization. It’s a reference for these published cases, with HTTP retrieval, caching, and partial user-agent matching outside its scope. Compare your production parser against the same cases, including the October 23 policy, before relying on it.
Expected summary: 18/18 fixture decisions match. Change one expected result to exercise the failure exit.
unzip robots-wildcard-fixtures.zip -d robots-wildcard-fixtures
cd robots-wildcard-fixtures
node run.mjsEncoding and group selection change the answer
The fixture treats /cafe and /%63afe as equivalent because c is an unreserved character. It keeps an encoded slash distinct from a path separator: /folder%2Fsecret does not become /folder/secret. Decode only the octets the protocol says to decode; decoding every escape can invent a directory boundary.
FixtureBot has two exact named groups whose rules combine. It doesn’t inherit the fallback group, so a request allowed for FixtureBot can be blocked for ExampleBot. The two-group worked example shows that selection before the paths are compared.
The runner gives normalization, group selection, matching, and precedence their own functions, so a disagreement with a production parser can be traced to the decision that changed instead of ending at one unexplained “blocked” badge.
A correct match can still use an expired policy
Add the exact deployed file to your release check, select the real crawler product token and evaluate a handful of representative paths. Include one intended block, one exception, a query variant, a capitalized path and a prefix neighbor. Preserve the returned policy when a release changes its effective result.
A crawler must also decide when that decision expires and what to do after a failed robots fetch. Keep the match result separate from those scheduling states in a crawl-frontier state machine. A correct string match against yesterday’s file can still produce the wrong request today.
Google’s open-source parser supplies another implementation and test corpus. I would put both the generator and production parser through the intended paths: the policy that matters is the one the next request will use.
Sources
- RFC 9309: matching and special charactersVerified Oct 26, 2026
Sections 2.2.1–2.2.3 cover groups, longest matches, encoding and special characters.
- Google: robots.txt specificationVerified Oct 26, 2026
Provider-specific group selection and path/query examples.
- Google’s open-source robots.txt parserVerified Oct 26, 2026
An independent production-oriented implementation and its test corpus.
Supporting material
- Executable wildcard fixture ↗Node 22+ runner, exact policy, 18 expected decisions and instructions. No dependencies or network requests.
- Expected wildcard decisions ↗Named inputs, selected group, expected permission and explanation.