Key takeaway
A conformal interval belongs to a specified calibration procedure and sampling assumption. Its marginal coverage statement does not guarantee coverage for each subgroup or deployed case.
Name the future-outcome claim before drawing the interval
A receiving report places an 80% interval around a point prediction and treats it as assurance for a particular facility. Ask how the interval was constructed, what data calibrated it and which probability statement is supported. A confidence interval for a population parameter, calibration of a binary probability and an interval for a future outcome answer different questions.
Angelopoulos and Bates describe split conformal prediction using scores from held-out calibration examples and a finite-sample quantile correction. Their stated theorem uses calibration and future examples drawn i.i.d. from the same distribution. This is a sufficient exchangeability setting. The coverage probability averages over the calibration sample and future example; it is not a guarantee conditional on every realised calibration set or subgroup.
Make the order-statistic convention visible
This hypothetical regression fixture fixes a point-prediction function before calibration. Define each score as the absolute residual |y−f(x)|. Nine invented calibration scores, sorted ascending, are 1,2,3,4,5,6,7,8,9 quantity units. Choose alpha=0.2 and explicitly use rank k=ceil((n+1)×(1−alpha)), with the kth smallest observed score when k≤n. There is no interpolation in this definition.
Here n=9, so k=ceil(10×0.8)=8 and the cutoff is 8. For an invented new point prediction of 50, the corresponding absolute-residual set is [50−8,50+8]=[42,58], with width 16 units. The author recomputed this arithmetic; no model was trained and no empirical coverage was measured. The displayed interval alone does not prove that this particular case has an 80% chance of falling inside it.
| Illustrative input or decision | Value | Consequence |
|---|---|---|
| Calibration scores | 1 through 9; n=9 | Preserve all nine ordered scores |
| Alpha and finite-sample rank | 0.2; ceil(10×0.8)=8 | Use eighth score under this rule |
| New point prediction | 50 units | Interval [42,58] units |
| Different alpha of 0.05 | ceil(10×0.95)=10 | No tenth finite calibration score; hold this finite fixture |
Separate fitting, conformalization and assessment
MAPIE’s documented split-regression workflow separates fitting the base regressor, computing scores on a conformalization set, and predicting intervals. Its prefit option makes the fitted-model boundary explicit. Retain the actual data roles, model version, score definition and receiving implementation. This article’s rank fixture is a mathematical specification; it does not claim that an arbitrary quantile API’s default interpolation reproduces it.
A model or tuning procedure that has already used the calibration outcomes needs a different justification. Likewise, grouped business episodes, temporal dependence or distribution shift cannot simply inherit the i.i.d. statement above. Examine whether the calibration and proposed future use genuinely supply the required exchangeability; a random-looking row order is not that evidence.
Check the finite-sample edge and the conditional claim
With only nine calibration scores, alpha=0.05 puts the specified rank at ten. Do not silently clamp it to nine and announce a 95% finite interval from the observed maximum. This illustration withholds that finite output. A real implementation needs a documented small-calibration policy, which may require more evidence or an appropriately specified unbounded set; no replacement finite rule is supplied here.
Ties and discrete scores can make a valid lower-bound coverage statement conservative. Do not relabel that as exact achieved coverage. A marginal statement also does not establish 80% coverage for every facility, rare operating condition or input value. Inspect relevant subgroup evidence and interval usefulness separately; a very wide set can satisfy a coverage objective while being unsuitable for the decision.
Keep the procedure attached to the receiving conclusion
The completed example accepts only the stated eighth-order cutoff and interval arithmetic under its declared convention. A receiving record retains training/calibration separation, eligible case unit, all calibration scores, alpha, rank and tie rule, score units, implementation version and limits on applicability. An independent assessment should report actual coverage and interval widths under a defensible design.
A claim about future outcomes must remain within the sampling and calibration boundary actually examined. No illustrative interval certifies deployment, safety, buyer demand or an archive’s licensing value. Use the precise procedure and its unresolved assumptions to decide whether another bounded evaluation is justified before approving access to any real sample.