Key takeaway
A probability ranking can be useful while its numbers remain poorly calibrated. Review the probability, outcome and decision separately.
Give the probability a precise outcome
A recipient may say that an archive supports failure prediction, then use an 80% model score to reserve maintenance capacity. Ask what that probability refers to: first failure within thirty operating days, any reported fault, or a confirmed remedy? The outcome window and eligibility rule must stay fixed before comparing predicted probabilities with actual events. Otherwise the apparent calibration changes when the label definition changes.
Scikit-learn describes calibration as agreement between predicted probabilities and observed positive fractions. Its calibration curve groups predictions and compares the mean probability with the fraction of positive outcomes. This is a different question from whether cases are ranked well or whether a particular cutoff produces acceptable actions. Record all three questions in the evaluation brief rather than treating one score as their answer.
Read the bins before approving the number
Consider this hypothetical held-out evaluation of two hundred invented records. Each row contains one hundred cases with the same prediction. The binary outcome is a confirmed failure within the agreed observation window, with complete follow-up. There are no missing outcomes, repeated units or fitted calibrator in this simplified example. Those assumptions would need examination in an actual archive.
For the 0.80 group, thirty failures and seventy nonfailures yield a binary Brier contribution of (30 × 0.20² + 70 × 0.80²) / 100 = 0.46. The 0.20 group contributes (20 × 0.80² + 80 × 0.20²) / 100 = 0.16. Equal group sizes give an overall Brier score of 0.31. These are reproduced toy calculations, not a licensing outcome or acceptable-quality standard.
| Predicted probability | Observed outcomes | Calibration gap | Review decision |
|---|---|---|---|
| 0.80 | 30/100 = 0.30 | Prediction exceeds fraction by 0.50 | Do not use 80% as an established failure rate |
| 0.20 | 20/100 = 0.20 | No gap in this bin | Agreement here alone does not validate other cases |
Separate calibration from other score changes
Brier loss measures more than calibration: the scikit-learn guidance explains that resolution and outcome uncertainty also enter the score. A lower overall loss can therefore coexist with worse calibration in a particular segment. Keep the bin counts, outcome frequencies and scoring convention beside the aggregate. Explain whether the positive label means failure and whether the binary score uses the 0–1 convention.
A few wide bins can hide disagreement inside a bin; tiny bins can produce unstable fractions. Prespecify bins and show counts rather than selecting attractive boundaries after seeing the answers. Review sites, equipment families and observation windows with the domain owner. A subgroup with very few events remains uncertain even if its plotted point sits on the diagonal.
Keep recalibration outside the final test
If recalibration is proposed, reserve separate data for fitting it and an untouched set for its final assessment. Do not fit the calibration mapping to the same outcomes used to announce success. Version the original model, fitted mapping, outcome specification and evaluated cohorts so a reviewer can distinguish a change in predictions from a change in the archive.
The decision record should say which probability uses are allowed, which are withheld, and what new evidence would change the hold. In the example, the team can investigate ranking while withholding capacity planning that assumes the 0.80 bin has an 80% failure rate. Use the diligence tool to request the missing evaluation design; an introduction, a later sample and any license still require their own recipient-specific decisions.