Can AI crawlers see your content? Check the response first

The comparison needs the browser’s own response

Another curl request may receive a different experiment, cookie branch, cache version or geographic response. Different bytes establish a delivery difference. They do not, by themselves, tell me what JavaScript changed.

For a source-to-DOM claim, preserve the main-document response from that browser navigation, along with the requested URL, redirects and final response. Keep its bytes, hash and decoding policy. A parser can later miss a heading that is plainly present in the retained file, and a decoding error can introduce replacement characters that resemble a content defect.

The saved Quotes sample makes a narrower observation. Both responses succeeded and both supplied a next-page address, but only one source contained quote-card elements. The runtime records in the other source explain why a render is worth checking; they are not a measured count of visible cards.

The browser has a point of view

Change the browser engine, viewport, cookies or consent state and the same URL can render a different page. Two runs become hard to compare when their records say only “desktop render” and omit the scenario or initial state. A repeatable run needs both.

An isolated context is a good default for an anonymous scenario. A continuing session is a different scenario, with its cookies and storage preserved deliberately. Service workers can serve old material, and interception can remove resources the real page would request. Even blocking analytics changes the experiment; I would record that intervention instead of quietly treating it as an untouched observation.

Collect HTTP responses alongside failed-request events. Playwright’s requestfailed event concerns a request that fails at the network level; a 404 or 503 is still an HTTP response. Listening only for requestfailed can leave the report clean while the article API returns an error. Playwright event semantics.

A quiet network can contain an empty article

A quiet connection can hide an empty shell after a failed article request, while polling can keep a finished article busy. Playwright discourages networkidle as a testing readiness criterion. Define completion through the article itself.

An error screen can have a perfectly good heading. A useful article check therefore identifies the intended document, then looks for its body, publication metadata and navigation within a deadline. Preserve the values observed and the condition that stopped the run. If only the heading arrived, that is a partial article capture; the unfinished body check should remain unresolved.

DOM presence doesn’t guarantee readable presentation: Playwright counts opacity:0 as visible. Inspect layout and appearance when the question is whether a person can read the article. For a source-canonical check, the returned head may be enough. I would choose the instrument from the claim.

The important difference is a changed field

A whole-document diff fills with generated IDs, wrappers and hydration markers. Begin with the fields relevant to the question: the heading and body, links and canonical, or the structured record. Mark each as added, removed, changed or unchanged, and link back to the source and DOM captures.

Keep original hrefs alongside resolved destinations and exact text alongside any normalization. Parsed structured records should be compared by stable identity, not JSON key order. These choices let normalization reduce noise without hiding the very change being investigated.

Consider a hypothetical hydration failure. The browser’s main response contains the article heading, a page exception occurs and the final DOM holds a loading shell. Together, those observations support an investigation of client-side loss. If the heading appears only in an independent HTTP fetch, they support a different investigation. Pairing the artifacts makes the diagnosis specific.

Re-extracting the saved response and DOM with one extractor holds the inputs fixed, so a changed report identifies an interpretation change; collecting the URL again changes the experiment and needs another observation with its own scenario, even when the fresh page looks healthy. Keep the failing pair to explain the original defect.

{
  "run_id": "render-2026-07-19-001",
  "requested_url": "https://example.com/article",
  "source_artifact": "sha256:source-hash",
  "dom_artifact": "sha256:dom-hash",
  "browser": { "name": "chromium", "version": "pinned" },
  "scenario": "anonymous-desktop",
  "completion": { "state": "partial", "missing": ["article-body"] },
  "failed_requests": 2,
  "field_diff_ref": "artifact://field-diff.json"
}

A title collision and a duplicate URL need different repairs. The canonical URL and duplicate title tag diagnostic traces title generation alongside redirects, split representatives, and conflicting URL signals.

The repair follows the failed stage

Find the first place where the intended body disappears. If the browser received it and the DOM lost it, investigate client mutation and runtime errors. If it never arrived in that response, investigate delivery. If both saved artifacts contain it while the report says missing, correct extraction before assigning work to the site. The paired files turn one symptom into different, narrower jobs.

The question determines where collection can stop. A source-canonical check may end with the returned head. An interaction or reading problem continues into execution and layout, where a DOM dump alone may be inadequate. Collecting more browser states is worthwhile when one of them can distinguish the remaining explanations.

After repair, repeat the same document identity, browser scenario, and completion conditions against the saved failing pair, because a healthier screen alone leaves open whether delivery changed, execution recovered, or the extractor simply learned to find content that was there all along. The explanation should survive after the symptom disappears.

For a public React/Vite site, the route release checklist carries this response-versus-render comparison through status codes, canonical URLs and production delivery.

The core answer belongs in the captured fields

Pick the content that makes the page useful: the opening answer, a product price, a table row or a link to another article. Look for those specific fields in the initial response and in the browser DOM. Keep their text and destination, rather than comparing only overall word counts.

Vercel’s December 2024 study reported JavaScript limitations for the agents it observed. Google documents a rendering stage in Search. A capability claim needs the product and observation date, because those products don’t necessarily collect the same page state.

If the public answer appears only after interaction or a client-side API call, serve it in initial HTML where appropriate, keeping private data protected. Then repeat the field check. I would make the answer and its working links available before measuring whether real bots fetch or cite them.