Key takeaway
Reordering one result array can preserve both marginal means while changing the variation of within-case differences. Check the join before interpreting a paired statistic.
Two equal-length arrays do not identify a pair
An evaluation compares two methods on the same cases, exports two score columns, then sorts one file independently. Both arrays have four values and the same means as before. A paired calculation will still run, but a difference can now join scores from different cases. The missing evidence is the shared case identity, rather than a matrix dimension.
SciPy’s ttest_rel is defined for related or repeated samples and computes a statistic from the mean of within-pair differences divided by their standard error. The pairing comes from the study and data alignment; the function cannot recover it merely because the arrays have the same shape. Keep that distinction in the receiving specification.
An unchanged difference of means hides the damage
This hypothetical fixture uses four invented case IDs. Method A scores are 10, 20, 30, 40; Method B scores on those same cases are 12, 19, 35, 46. Define each difference as B−A. Correct case matching gives 2, −1, 5, 6: mean 3, sample standard deviation √10≈3.162278 and a sample standard error of the mean difference≈1.581139. The calculation assumes independent paired cases only for this teaching illustration.
If B is reversed while A retains its order, the differences become 36, 15, −11, −28. Their mean remains 3, but their sample standard deviation is≈28.225284 and standard error≈14.112642. The author recomputed both arrays locally. Neither is a real evaluation, and the corrupted pairing has no valid case-level interpretation even though its summary arithmetic can be computed.
| Case ID | Method A | Correct Method B | Correct B−A | B after independent reversal |
|---|---|---|---|---|
| C1 | 10 | 12 | 2 | 46 |
| C2 | 20 | 19 | −1 | 35 |
| C3 | 30 | 35 | 5 | 19 |
| C4 | 40 | 46 | 6 | 12 |
A different test option cannot repair a missing match
SciPy documents ttest_ind for independent samples; its equal_var option selects between equal-population-variance and Welch forms. That option does not declare paired identities. Conversely, choosing ttest_rel does not establish that the receiving join is correct. Decide the sampling model and intended comparison before selecting a function.
Keep a common case key, method/run identifier and outcome field in one aligned comparison record. Verify a one-to-one join for the intended case unit, then inspect mismatched and duplicated keys. If one method lacks a result, retain that missing paired outcome and the exclusion rule. Independently dropping missing rows from each column can create another positional mismatch.
Separate alignment from inferential assumptions
This fixture demonstrates a change in the variation of differences; it does not endorse a significance decision from four cases. A statistical reviewer must assess the sampling, independence between pairs, distributional assumptions and intended inference. Repeated observations from the same episode can also need a dependence-aware method even when each pair is correctly joined.
Prespecify direction, units, eligible cases, treatment of failed runs and any uncertainty method. Preserve the actual paired differences, including uncomfortable cases. An aggregate difference of means cannot prove the pairing survived, and a large sample count cannot replace the evidence linking the two outcomes to the same observational unit.
Accept the case map before accepting the comparison
For these invented values, the receiving contract requires differences 2, −1, 5, 6 attached to C1 through C4. The reversed array fails that identity check. A useful acceptance note records the fixture IDs, original rows, join keys, matched/unmatched counts and the expected differences before a report interprets uncertainty or a p-value.
If an earlier result used the wrong alignment, preserve its run identity and correct the comparison explicitly. Do not silently replace the numbers while retaining an unchanged analysis label. The resulting evidence concerns a reproducible comparison procedure; it supplies no buyer demand, model deployment approval or permission to disclose the underlying cases.