Key takeaway
A readable string can differ from the source bytes, and an equal-looking string can contain different character sequences.
Decide what must remain identical
An export passes a visual spot check: names have accents, notes look readable, and identifiers appear unchanged. That does not establish byte equality or character equality. Define which representation the receiving use needs. A source archive may require exact original bytes for traceability, while a search key may deliberately use a normalized derivative. Keep those products distinct rather than overwriting the original.
The Unicode Consortium describes reversible encoding for valid Unicode sequences and the information loss that can follow invalid-sequence replacement. Its normalization specification separately describes canonical and compatibility equivalence. Encoding converts character representations; normalization can change their sequences. Neither step supplies the missing original character when an earlier process has already replaced it.
Two spellings that look the same
This hypothetical fixture contains two invented labels that both display as café. One ends in precomposed U+00E9; the other ends in U+0065 followed by combining U+0301. Their original code-point sequences differ. NFC normalization makes these two labels canonically comparable, but the normalized derivative is not evidence that their original bytes matched.
The fixture also contains superscript notation m². A compatibility normalization such as NFKC changes the superscript 2 to an ordinary 2. That might be useful for a separately specified search key, but it can remove a distinction in technical notation. Unicode explicitly cautions against blindly applying compatibility forms. The example’s acceptance rule preserves original notation and keeps any search derivative in another field.
| Fixture value | Source distinction | Candidate transformation | Decision |
|---|---|---|---|
| café, composed é | U+00E9 | NFC derivative | Keep source and derivative separately |
| café, combining accent | U+0065 + U+0301 | Same NFC derivative as above | Do not report equal original bytes |
| m² | Superscript exponent | NFKC becomes m2 | Hold unless this semantic loss is intended |
| Replacement character � | Original character unknown | No reliable reconstruction | Record loss; do not guess |
Test the actual decoding boundary
Record declared encoding, decoder version, invalid-sequence policy and whether a byte-order mark was present. Preserve the original input when authorized. Run a fixture through the actual export and receiving reader, then compare the intended invariants: byte hash for unchanged files, code points for text preservation, and approved normalized keys for search. A checksum of the transformed file cannot establish that no source characters were lost before it was created.
Include accents, combining marks, non-Latin scripts and technical symbols relevant to the proposed use. Test long identifiers and round trips through the receiving CSV or database application. Report replacements and decode errors by field and count. An application that substitutes a replacement character may still produce valid output text, so “the file parsed” is an insufficient acceptance rule.
Separate matching convenience from record evidence
A useful export contract names the source-text field, normalized derivative, chosen form, version, case treatment and collision policy. If two source identifiers become the same derivative, do not merge their records automatically. Keep the source identity and ask the owner whether the apparent match represents one entity, different entities or an unresolved ambiguity.
The fixture supports a bounded conclusion: the two café labels can share an NFC search representation, while the original sequences remain distinct and m² remains preserved in the source-text field. It does not establish person identity, translate the text or repair damaged characters. The inventory builder can record unresolved encoding fields; the readiness planner can assign round-trip checks. Original content, derivative text and any later sample require their own authority review before disclosure.