Key takeaway
A selected winner needs the history of the selection. One attractive comparison cannot stand in for all the attempts.
Define the family of claims before selecting a winner
A buyer tests twenty model configurations and reports that one beats a reference at p < 0.05. Ask how many comparisons were tried, which metric was chosen, and whether the same test set informed the search. A p-value belongs to a specified statistical test and assumptions; selecting the most attractive result changes the review problem.
R’s official statistics documentation distinguishes adjustments that control family-wise error from false-discovery-rate methods. Those goals are not interchangeable. The method, comparison family and dependency assumptions need an analyst’s review. This guide supplies a trial ledger and a simple Bonferroni illustration, not a universal claim that this conservative method is the best choice.
An attractive raw result that does not survive its family
In this hypothetical review, the team declared twenty comparisons in advance, all using valid underlying tests under their specified null hypotheses. The family-wise target is 0.05. A simple Bonferroni comparison uses 0.05 /20 =0.0025 for each test; equivalently, multiply each raw p-value by 20, capped at 1. The invented ledger shows three of those twenty outcomes, not a three-test family.
Under the additional toy assumption of independent tests with all twenty null hypotheses true and each test having exactly a 0.05 null rejection probability, the chance of at least one raw p < 0.05 is 1 − 0.95²⁰, approximately 64.15%. Real evaluation tests may be dependent, so that number is not an estimate for a buyer’s actual trial run. The adjusted-p calculations below do not require adopting that independence illustration.
| Comparison | Raw p-value | Bonferroni-adjusted p | Decision at family target 0.05 |
|---|---|---|---|
| Candidate A | 0.030 | 0.600 | Does not pass this illustrative rule |
| Candidate B | 0.001 | 0.020 | Passes the arithmetic rule, subject to valid test design |
| Candidate C | 0.004 | 0.080 | Does not pass this illustrative rule |
Separate tuning evidence from final assessment
An adjustment does not fix test-set leakage. Scikit-learn explains that non-nested selection can produce optimistic results when tuning and evaluation reuse the same evidence. In nested cross-validation, the inner procedure selects parameters and the outer assessment evaluates that selection procedure. It is not a permission to keep inspecting an external final set until a favorable result appears.
Record model/configuration identifier, metric, comparison purpose, data split, time of result inspection and whether the outcome changed the next trial. Include failed and abandoned attempts relevant to the declared family. A replacement metric chosen after seeing results needs disclosure. If the history is incomplete, describe the result as exploratory and arrange independent confirmation rather than reconstructing a smaller family around the winner.
Keep practical improvement beside statistical evidence
A statistically adjusted result can still be too small to matter operationally. Report the effect size, units, reference behavior, evaluation population and uncertainty appropriate to the design. Do not replace those details with an adjusted-p badge. Conversely, failing this illustrative rule does not prove two systems are identical; it limits what this evidence supports.
The useful decision record names the declared family, selected analysis, untouched confirmation boundary and unresolved assumptions. Candidate B in the example passes only the stated arithmetic screen, not buyer diligence, deployment approval or a commercial acceptance milestone. Use the diligence builder to request the full experiment history. A named introduction, access to any real sample and a later license remain separate permission and contract decisions.