Key takeaway
A relevant sentence can become a wrong answer when its exception, heading or table context disappears during chunking.
Trace an answer back to its complete context
A buyer retrieves a paragraph from a service manual and asks whether it answers the evaluation question. Before scoring the answer, inspect what was excluded when the source was divided into chunks. A permissioned document can still yield a misleading excerpt. The decision here concerns semantic context within the permitted document set; access boundaries and untrusted instructions require their own controls.
Microsoft’s chunking guidance distinguishes fixed-size, variable and semantic approaches. Its architecture guide warns that small chunks can lose context spanning several pieces. Those sources support testing the relationship between document structure and retrieval, not a universal number of characters. Character length also is not automatically the model’s token count.
A synthetic manual that changes the answer
This hypothetical manual says: “Replace the gasket after 500 operating hours.” The following sentence says: “For the high-temperature variant, replace it after 200 hours.” A heading identifies the model, and a table footnote excludes shutdown-storage time from operating hours. The invented evaluation question asks about the high-temperature variant after 250 operating hours.
A chunk containing only the first sentence appears relevant but omits the exception. For this example, the review target is an answer that acknowledges the 200-hour condition, cites its location and keeps the operating-hour definition. The table records evidence lost by three candidate chunk designs. It is not actual maintenance advice or a claim that any real archive contains these passages.
| Retrieved design | Context retained | Observed toy answer risk | Review action |
|---|---|---|---|
| First sentence only | General interval | Misses high-temperature exception | Reject this chunk for the question |
| Both sentences, no heading | General and variant interval | Cannot reliably establish model scope | Add model path and source location |
| Rule, exception, heading and footnote link | Scope and hour definition | Evidence supports the intended toy answer | Verify the link resolves in retrieval output |
Use a context ledger, not a length contest
Record source document version, heading path, chunk identifier, parent location, table headers, footnotes and any linked exception. Keep enough structure for the reviewer to reconstruct the passage. If a table row loses its unit or column label, a numeric answer can be fluent and wrong. If a sentence depends on “except as above,” retain or resolve the referenced text rather than letting that phrase become an orphan.
Overlap is one possible continuity aid described in Microsoft’s guidance, but it does not guarantee that a distant exception or footnote will be retrieved. Test the actual candidate passages for the chosen query set. Report when the expected context sits outside the returned material, even if the original document contains it. A larger chunk can also bring irrelevant content, so judge answer support and source traceability together.
Review the pipeline before claiming evaluation success
Create a small synthetic fixture with a rule, an exception, a table with units and a revision note. Inspect the converted text, final chunks, retrieved set and answer separately. This distinguishes a parsing loss from a chunk boundary mistake or a retrieval ranking failure. Retain the version of each step and the question wording so a later configuration change can be compared against the same evidence.
Accept only a bounded statement about the inspected question set and document structures. Do not infer that preserving this fixture validates all procedures, permits a broader document search or proves buyer demand. Use the diligence builder to ask for the context ledger and the readiness planner to assign unresolved parsing work. Later access to real records, an evaluation sample and any license remain distinct permission decisions.