Technical SEO
Crawler directives: the right control for each outcome
Use robots.txt to control cooperative crawling, noindex to keep a public page out of Search, and authentication to protect a private page. A crawl block alone can leave the address in search results.
To keep a public page out of Search, serve a readable noindex. To protect a private invoice, require authorization. To reduce cooperative crawling, use robots.txt. These controls act at different points, which is why a crawl block can leave a URL indexed and an indexing directive can leave private content exposed.
The table starts with the result you need, then shows the response that can deliver it. Canonicals nominate equivalent representatives; supported robots meta and X-Robots-Tag rules govern indexing or presentation after a fetch. The example.com paths are teaching examples.
Method & limits
Provider documentation checked October 26, 2026. The response designs and local-host demonstration are authored examples, not measured behavior by a search engine or AI provider.
Which control owns the result?
A public thank-you page, a private invoice and a duplicate campaign URL can all be described loosely as “do not index this.” Their correct delivery is different. The invoice should never become public; the thank-you page can expose noindex; the campaign URL may belong to the same canonical family as a clean guide.
| Desired outcome | Control and location | Keep fetch available? | What to verify |
|---|---|---|---|
| Reduce crawling of low-value paths | Disallow in /robots.txt | The blocked paths are intentionally not fetched. | Selected group and representative path/query matches. |
| Exclude public HTML from Search | noindex in its head or X-Robots-Tag header | Yes: the indexer must read it. | Returned directive and later indexed state. |
| Exclude a public PDF | X-Robots-Tag: noindex on the PDF response | Yes: an HTML tag cannot be embedded in its head. | Actual PDF GET headers across CDN and origin. |
| Limit search snippets | Supported nosnippet or preview controls in meta/header | Yes. | Correct resource scope; this is not a crawl block. |
| Consolidate equivalent URLs | rel=canonical or an appropriate permanent redirect | Keep the intended representative accessible. | Equivalent content and aligned links/sitemap. |
| Deny private access | Authentication plus resource authorization and safe caching | Only authorized principals receive the content. | Anonymous and wrong-user body checks. |
| Protect origin capacity | Rate/concurrency limits and caching; provider pacing where supported | Apply a deliberate service policy. | Request rates, statuses and expensive origin work. |
The response chooser shows each control in place
Choose a case to see its policy and response together. These are independent alternatives, not a single configuration to combine. The public-HTML case deliberately leaves crawl access open so noindex is visible.
For Google, robots meta and X-Robots-Tag can deliver the same supported indexing rules. The robots meta specification defines their scope, user-agent targeting and conflict behavior. If one delivered rule permits indexing but another says noindex, adding more permissive markup does not cancel the restrictive rule.
Allow the fetch; return noindex.
GET /robots.txt → 200
User-agent: *
Disallow:
GET /thanks → 200
<head><meta name="robots" content="noindex"></head>Verify the directive in the response, then Google’s later indexed state.
A directive governs the response that carries it
A robots file governs its serving scheme, host and port. A meta element belongs in the HTML head of the page it controls. An X-Robots-Tag travels with the resource’s HTTP response, which makes it useful for PDFs and other non-HTML content. None of these locations automatically governs every hostname or alternate download route.
Capture the final GET response, including headers and initial HTML. A CMS setting or HEAD request can miss what a visitor receives. If JavaScript changes the head, inspect the rendered document too, as the HTML and DOM guide shows.
For link following, distinguish a page-level nofollow hint from a robots fetch restriction. For AI collection and search, identify the provider and purpose before choosing a token. The separate AI crawler policy hub owns that purpose-specific decision; a training opt-out is not a universal Search exclusion.
Combined controls depend on the fetch sequence
A crawl block plus noindex is a broken sequence when the block prevents the indexer reading noindex. The indexed-though-blocked repair shows the corrected public response. Canonical plus noindex needs a separate consolidation-versus-exclusion decision.
For capacity, publishing Crawl-delay is a request to supporting clients; it does not impose a server limit. Use the provider-support comparison before deciding whether it helps your actual traffic.
Match acceptance to the intended result: the returned header shows delivery, a later Search Console observation shows what Google processed, and anonymous and wrong-user tests check access on the tested routes and cache states. Choose the control that can deliver the result, then verify that result.
For a cooperative fetch rule, the general robots.txt matching examples show which path a rule permits or blocks.
Sources
- Google: robots.txt introductionVerified Oct 26, 2026
Crawl policy, indexing limitations and protection alternatives.
- Google: robots meta and X-Robots-TagVerified Oct 26, 2026
Directive delivery, resource scope and restrictive-rule combinations.
- Google: canonical URL methodsVerified Oct 26, 2026
Representative selection for equivalent content.
- RFC 9309: security considerationsVerified Oct 26, 2026
Robots exclusion is not access authorization.