Operational article · published
Run a Crawlability Audit Before Diagnosing Indexing
Diagnose missing Google pages by checking discovery, HTTP responses, rendered content, robots directives, and canonical selection before changing the site.
Reviewed 2026-11-18 · Google guidance and indexing diagnosisThe task and the failure mode
Built for: Site owners and developers investigating a valuable page that is absent from Google, including React sites whose browser view differs from the server response.
A missing search result is a symptom, not a diagnosis. A URL can be unknown to Google, known but unfetched, fetched with an error, rendered without its answer, or treated as a duplicate. Each state calls for a different response. Repeatedly submitting the same URL cannot tell you which layer failed. Start with the exact URL and preserve what the server and Search Console actually report.
Decision brief
Use the Page indexing report to choose a URL, then inspect that URL individually. Discovered — currently not indexed means Google knows the address but has not crawled it. Crawled — currently not indexed means a crawl occurred without current index inclusion. Neither label, by itself, identifies the underlying cause.
Separate the indexed inspection result from Test Live URL. The first describes Google’s recorded state; the second checks the page available now. A release can fix today’s response while the recorded crawl still predates it. Keep both timestamps rather than declaring that one report must be wrong.
Approve a repair when you can name the failed layer and show the expected replacement behavior. Fix access before rewriting prose for an inaccessible page. When access and canonical signals are sound, compare the page’s actual answer with nearby pages before deciding whether to improve it or consolidate genuine duplicates.
Questions to answer before changing the system
- 01Which exact URL should appear, and does its purpose differ from the other pages on this site?
- 02What does URL Inspection report, and when did the recorded crawl occur relative to the last deployment?
- 03Does an unauthenticated GET return the intended page directly, rather than a redirect, error, or empty shell?
- 04Are the main answer, canonical URL, and navigational links present before and after JavaScript runs?
- 05Does Google select a different canonical, and do internal links or sitemap entries support that competing URL?
Workflow
- 01Capture the starting state. Record the requested URL, importance, expected canonical, Search Console reason, and last crawl time in the downloadable worksheet. Keep redirects and intentionally removed pages in separate groups so the excluded total is not mistaken for a list of broken pages.
- 02Fetch without browser authentication. For a site you control, curl -sS -D headers.txt -o page.html https://example.com/service saves the initial GET headers and body. Inspect the status and Location header before following redirects. A successful HEAD request alone does not establish what a crawler receives in the body.
- 03Trace any redirect chain. Record every hop and its destination, including protocol, hostname, case, and trailing-slash changes. A permanent move to an equivalent page is legitimate. A service URL redirected to an unrelated homepage is not a satisfactory repair merely because the final response is 200.
- 04Read robots.txt and both indexing surfaces. Check the applicable user-agent group, the HTML robots meta tag, and every X-Robots-Tag response header. A robots.txt disallow can stop Google seeing a noindex directive. Remove an accidental block only after confirming the page is intended to be public.
- 05Compare page.html with a fresh browser rendering. Look for the actual answer, one clear H1, the intended canonical, and real anchor href links. Then test a direct visit to the deep URL rather than only clicking from the homepage. A working client-side transition can conceal broken direct routing.
- 06Compare the declared canonical with Google’s selected canonical where inspection provides it. Inspect both destinations and their content. Repair contradictory links, redirects, or metadata when the intended representative is supported by the content; do not force unrelated pages to share a canonical just to reduce report counts.
- 07Re-run the failing check after deployment and retain the release identifier. For an important materially changed page, use the supported recrawl request once when appropriate. Check later inspection results against the deployment date. Report the technical defect as repaired separately from any subsequent Google indexing decision.
Evidence to retain
- The original requested address and the intended canonical, stored separately rather than normalizing away the discrepancy under investigation.
- GET status, redirect hops, response headers, and the saved HTML body, with a timestamp and the environment tested.
- The applicable robots.txt rules plus HTML and header-level indexing directives, including any disagreement between those controls.
- A raw-versus-rendered comparison of the primary answer, heading, canonical, and links, not simply a successful screenshot.
- Search Console’s reason, selected canonical when available, and last crawl date; mark unavailable fields as unknown rather than empty success.
- At least one relevant internal linking page and the sitemap entry, showing which URL each actually names.
- The post-deployment repetition of the failed test and an unchanged control page, tied to the released commit.
Hypothetical case: a working browser page with the wrong server response
- Observation
- A repair guide opens correctly after a homepage click, but a direct GET to /guides/leak-repair returns the homepage shell. Its canonical also names the homepage.
- Diagnosis
- The deployment serves a generic SPA fallback at an intended public route. The browser later loads the guide, creating different server and rendered crawl surfaces.
- Change
- Generate or server-render the guide at its own URL, with its answer and self-canonical in the initial response. Preserve the homepage at its existing address.
- Acceptance
- Direct GET now returns the guide; rendering keeps the same answer and canonical. A nonexistent guide returns a genuine missing response, not the working guide shell.
- Remaining uncertainty
- These tests prove the changed delivery behavior. They do not prove Google has crawled the release or chosen to index the guide.
Checks before closing an indexing repair
- The target is a public page with a distinct reader task.
- Its direct GET returns the intended content and HTTP status.
- Every redirect has an equivalent, reachable destination.
- No accidental robots block prevents fetching the target.
- No inherited HTML or header directive excludes an intended search page.
- The useful answer survives a direct visit and browser rendering.
- Canonical signals agree with the intended content representative.
- Internal links and sitemaps name the preferred public URL.
- A real missing URL still behaves as missing after the repair.
- Deployment verification is recorded separately from Google’s processing state.
What to measure—and what it does not prove
- Track intended public URLs with verified direct responses as a technical measure. Name the denominator; a five-page sample is not a complete site audit.
- Track inspected URLs by their recorded Google state and crawl date. Keep not inspected separate from not indexed to avoid manufacturing an exclusion rate.
- Measure newly observed crawls after the release date before attributing a later indexing change to the repair. Other changes may have contributed.
- Review impressions and qualified enquiries separately. A page can be indexed without meaningful visibility, and visibility does not establish commercially useful traffic.
Boundaries and caveats
Changing a user-agent string locally does not reproduce Googlebot’s network, rendering environment, or previously recorded index state.
A 200 response is necessary for many intended search pages but is not proof of useful content or index inclusion.
Do not label a valid redirect, correct alternate canonical, or deliberately removed page as a defect simply because it is excluded.
After access is sound, content review still matters. Replace repeated abstractions with an answer, concrete examples, and evidence specific to the page’s question.
Primary sources
- Google Search Console: Page indexing reportsupport.google.com
- Google Search Console: URL Inspection toolsupport.google.com
- Google Search Central: JavaScript SEO basicsdevelopers.google.com
- Google Search Central: Consolidate duplicate URLsdevelopers.google.com
- Google Search Central: Block indexing with noindexdevelopers.google.com
Start with one bounded case
Choose the most valuable missing URL and complete one worksheet row using a direct GET and URL Inspection. Fix the first demonstrated failure, retain the before-and-after response, and then test another URL using the same template. Expand the repair only when that second observation supports the same explanation. If the direct response and browser disagree, use the raw HTML and rendered DOM audit above. If those agree and previously indexed URLs are dropping out across a template, use the indexing regression playbook to scope the incident.