Key takeaway
Identical data files do not guarantee identical execution. Save the environment and the allowed variation too.
Separate data identity from execution identity
A reviewed package can preserve its source files and still produce a different result when a dependency or container changes. Docker explains that image tags are mutable; a digest identifies an exact image version. pip distinguishes version pinning, downloaded-package hash checks and bundled compiled dependencies. Those controls address different parts of an execution environment.
Start with the question a receiving reviewer needs to repeat. Name the task, input package, command or procedure, output metric and permissible variation. If the evaluation depends on a hosted API whose internal version cannot be fixed, disclose that limit rather than presenting a local environment file as complete reproduction evidence.
A completed environment record
This hypothetical review compares two invented runs. Both use data package v3. Run A records its dependencies, image digest and architecture. Run B records only a mutable image tag and allows new dependency versions. No real digest or benchmark score is supplied.
| Environment element | Run A | Run B decision |
|---|---|---|
| Data package | v3 with verified file identity | Same v3; data identity agrees |
| Container image | Exact digest retained privately | Tag only; resolve exact image first |
| Dependencies | Direct and transitive versions plus hashes | Unbounded updates; recreate a fixed set |
| Platform | OS and CPU architecture recorded | Platform missing; replay claim held |
| Output acceptance | Metric and tolerance set before replay | Tolerance absent; cannot decide equivalence |
Pin intentionally, then maintain the pinned set
pip’s documentation notes that transitive dependencies matter and that a wheelhouse can be specific to an operating system and architecture. Record more than the top-level package list. Keep the permitted installation source, version set and artifact verification method with the run. Do not insert a made-up hash to make the record look complete.
Docker also notes the update tradeoff: retaining an exact image means updates need an intentional process, including security fixes. Preserve the old environment for bounded comparison where appropriate, review the update, and issue a new execution identity for the next accepted environment. Reproducibility should not become an instruction to ignore vulnerabilities indefinitely.
Describe variation before seeing the rerun
Some tasks have legitimate numerical or sampling variation. Agree whether acceptance requires identical bytes, identical decisions on named cases, or a metric within a justified tolerance. State random-seed handling and known external dependencies when relevant. A fixed seed alone cannot establish identical behavior across every runtime and platform.
Use a synthetic replay case that includes the normal path and at least one meaningful boundary. Compare the observed difference with the tolerance agreed beforehand. If the difference changes a buyer decision, investigate it rather than expanding the tolerance after seeing the result. Save the run identifiers and the observed values so the disagreement can be examined.
Accept the reproducible part, disclose the rest
A useful handoff names what can be reproduced locally, what needs the receiving system and what remains dependent on an external service. Ask the recipient to retain environment identity with its conclusions. The package-version guide describes the source bytes and transformations; this environment record explains the execution that produced a result from them.
Use the diligence builder to request the relevant artifacts and the readiness planner to assign missing environment tasks. This procedure does not certify performance, guarantee future replay or establish rights to distribute software dependencies. A synthetic environment test can precede a real sample, whose permission and security conditions require a separate decision.