Technical SEO
Indexed, though blocked by robots.txt: inclusion or removal?
A crawl block can stop Google reading a page while its address remains in Search. The fix depends on whether the address belongs there.
To fix “indexed, though blocked by robots.txt,” decide whether the URL belongs in Search. For inclusion, remove the unwanted crawl block. For removal of a public page, let Google fetch noindex. The same warning leads to opposite repairs because Google can index an address discovered through links without reading its page.
Google’s Page indexing report describes that mismatch. Private content needs server-side protection before either search decision.
Method & limits
Primary documentation checked October 26, 2026. The example.com responses and local fixture are authored teaching examples; they do not report a Google indexing result or observed provider compliance.
The intended outcome determines the repair
A product accidentally swept into a directory block needs crawl access restored. A public confirmation page that should stay out of Search needs a readable noindex. Applying one removal recipe to every URL in the warning can hide the very product you want customers to find.
For a private invoice, protect the representation with authentication and authorization first. Making it publicly readable so a crawler can see noindex would create the wrong result. The general robots guide separates permission checks from search controls.
| Desired outcome | Robots/page response | Verification that matters |
|---|---|---|
| Searchable product | Allow the product fetch; return a useful 200 with no exclusion directive. | Live inspection can fetch it; indexed inspection later reflects the intended URL. |
| Public page excluded from Search | Allow the fetch; return 200 and noindex in the HTML head or HTTP header. | Google can read noindex; a later indexed observation shows exclusion. |
| Private invoice | Deny unauthorized access at the server and shared caches. | An anonymous request never receives invoice contents, including after a signed-in visit. |
The same noindex needs a reachable response
In the first example, /receipts/thanks is disallowed, so Google can’t read the noindex in its template. In the repaired public-page response, the policy permits the fetch and the returned HTML carries the exclusion.
The fixture uses an exact-path exception so other /receipts/ URLs stay blocked. The exception does not cover query variants: /receipts/thanks?order=123 remains a different match. List the real variants, decide their responses, and test them before copying the rule. Never put customer identifiers in a public teaching fixture.
The directive must cross the crawl boundary.
- 01
Blocked
Google knows the URL from a link. The robots rule prevents reading its noindex.
- 02
Readable
The exact path is allowed. Its 200 response exposes noindex to the crawler.
- 03
Observed later
A later Google crawl can process exclusion. A live test alone does not establish it.
The public policy and the blocked page are two responses in the fixture.
GET /robots.txt → 200
User-agent: *
Disallow: /receipts/
GET /receipts/thanks → 200
<head><meta name="robots" content="noindex"></head>The longer Allow applies only to this exact path. Other receipt paths retain their crawl rule.
GET /robots.txt → 200
User-agent: *
Disallow: /receipts/
Allow: /receipts/thanks$
GET /receipts/thanks → 200
<head><meta name="robots" content="noindex"></head>Delivery changes before Google’s stored state
First capture the deployed /robots.txt response for the exact scheme, host and port of the affected URL. Record the final GET status, redirect chain and body. Evaluate the selected crawler group against the actual path and query; a change to the wildcard group may leave a specific Googlebot group untouched.
Next fetch the page. Inspect its X-Robots-Tag header and HTML head, then the rendered document if JavaScript changes the head. For a PDF, use the HTTP header instead of an HTML tag. Google’s noindex documentation requires crawl access for either method.
Run URL Inspection’s live test on the same URL and inspect the returned content. Then compare the last crawl and indexing state in indexed inspection. Google documents the distinction: the live test shows current delivery, while indexed inspection shows its stored observation.
When fresh public delivery is correct but Google’s stored crawl predates it, keep the response stable while the next fetch catches up, because another robots edit can delay the very noindex you need Google to read. A recrawl request and its eventual effect need their own dates.
Future crawls still need noindex
Keep noindex readable after the page disappears. Future crawls need to encounter it for the public URL to stay excluded. A deleted document can return an appropriate not-found or gone response; urgent search-result removal uses a temporary removal workflow.
If the real objective is to consolidate equivalent pages, use the canonical and noindex decision before choosing exclusion. A canonical points to a representative; noindex removes this URL. Those are different publishing intentions.
The repair succeeds when the response serves the intended outcome and Google’s later crawl reflects it. Until then, a stable, readable directive is a better next move than another policy change.
Sources
- Google Search Console: Page indexing reportVerified Oct 26, 2026
The indexed-though-blocked warning and separate inclusion/removal actions.
- Google Search: noindex implementationVerified Oct 26, 2026
The directive must be fetchable; noindex in robots.txt is unsupported.
- Google Search Console: URL InspectionVerified Oct 26, 2026
Live tests and indexed observations answer different questions.