Key takeaway
A failure to detect a difference can reflect imprecision. Equivalence asks whether the evidence rules out differences outside a justified, predeclared margin.
Write the acceptable difference before reading the result
A recipient compares a new extraction method with a reference and reports no statistically significant difference. The owner interprets that sentence as evidence that the methods are interchangeable. Ask how large a difference would matter for the actual use. A test of a zero difference and an assessment of practical equivalence address different claims, even when they use the same observations.
The American Statistical Association’s p-value principles explain that significance does not measure effect size or practical importance and that decisions should not rely only on passing a threshold. The statsmodels documentation for two one-sided tests describes a separate equivalence test with specified lower and upper bounds. Neither source supplies a universal acceptable margin for an operational dataset.
Three hypothetical intervals, three different conclusions
For this illustrative interpretation exercise, assume an analyst has selected a valid independent-samples design and its appropriate interval-based equivalence procedure in advance. The outcome is measured in the same invented units for both methods. The predeclared acceptable difference is strictly between −5 and +5 units. The table supplies fictional 90% confidence-interval summaries for that chosen procedure, not raw observations or a newly executed test.
At the familiar two-one-sided-test level of 0.05, the corresponding two-sided interval criterion uses 90% confidence. An ordinary two-sided difference test at 0.05 instead uses its 95% interval. Do not swap those levels after seeing results. The table compares interval placement against the margin; it does not calculate p-values, establish a real equivalence result or certify the example’s assumed analysis design.
| Invented difference and 90% interval | Relation to ±5 margin | Equivalence reading under stipulated procedure |
|---|---|---|
| Estimate 2; interval −18 to 22 | Includes 0 and extends far beyond both margins | Too imprecise to establish equivalence |
| Estimate 2; interval 1 to 3 | Entirely inside −5 to 5 | Supports this bounded equivalence criterion |
| Estimate 8; interval 6 to 10 | Entirely above the upper margin | Does not support equivalence |
Read a wide interval as an unresolved decision
The first row is compatible with a range of substantively different effects, including zero. A non-significant difference test would not turn that imprecision into proof of close agreement. More informative evidence may require a better measurement design, reduced noise or a planned sample with enough precision. Do not promise that merely adding an arbitrary number of rows will solve it; dependency, outcome validity and coverage can remain limiting.
The second row illustrates another distinction: a small effect can be estimated away from zero yet remain inside a justified practical margin. Statistically detectable and operationally acceptable need not be opposites. Report the estimate, units, margin and interval procedure together. A badge saying same performance would erase the bounded meaning of the conclusion and invite reuse for another outcome or workflow.
Use a test that matches the observations
The cited statsmodels function is specifically documented for two independent samples. Comparing two extractors on the same records creates a paired structure, so that function is not automatically the right implementation. Have the analyst choose an appropriate procedure for the actual pairing, repeated episodes, variance assumptions, missing outcomes and intended population. The numerical illustration does not authorize ignoring those features.
Also justify the margin from the intended decision, rather than deriving it from the result that happened to appear. A transcription error tolerance, turnaround-time difference and dangerous failure rate require different substantive judgments. Record who approved the margin and why, before the final evidence is examined. If it changes later, disclose the change and reassess the analysis rather than presenting the revised criterion as original.
Make the acceptance statement as narrow as the evidence
The completed hypothetical review withholds an equivalence claim for the wide first interval, supports only the stipulated bounded criterion for the second and rejects that criterion for the third. No statement says the methods are identical. An actual acceptance note should identify the population, outcome, margin, design, interval/test method and remaining limitations, with the responsible statistical reviewer’s conclusion.
This question sits beside, rather than replaces, multiplicity control and small-sample uncertainty. Correcting many p-values does not choose an equivalence margin. A tight interval on an unrepresentative sample does not establish agreement everywhere. Use the bounded result to decide whether the particular evaluation claim is ready, instead of converting no significant difference into an unrestricted product warranty.