An earlier reading in this review argued that a model's confidence is the wrong kind of information for deciding whether to act on its output. That argument deserves to be tested against the machine-learning field's most serious attempts to measure its own uncertainty, because if any of them produced a genuine warrant for acting on a particular claim, the argument would fail. This entry walks that apparatus, from the crude to the rigorous, and finds that every rung answers a different question than the one this work asks.
Confidence is not even one thing
The first and most important refinement comes from the distinction, now standard in the field, between two kinds of uncertainty. Aleatoric uncertainty is noise inherent in the observations, the kind that cannot be reduced even if more data were collected. Epistemic uncertainty is uncertainty in the model itself, the model's ignorance about which parameters generated the data, and it can be explained away given enough data (Kendall and Gal, 2017). These are not two labels for one quantity; they are different in origin and behaviour, and a single confidence score conflates them. This sharpens the earlier argument considerably. A model's confidence is not merely an imperfect measure of whether its assertion is true; it is a blend of noise in the world and the model's ignorance of itself, and neither of those, however precisely quantified, is evidence that the asserted fact holds.
The paper that formalized this distinction for modern systems opens with a death. The first fatality from an assisted driving system, it notes, was caused by the perception system confusing the white side of a trailer for bright sky (Kendall and Gal, 2017). The system conveyed a state, no obstacle ahead, that was indistinguishable, to it, from the truth, and that state was acted upon at speed. The authors' own conclusion is the one this work builds on: had the system been able to represent what it did not know, it might have made a better decision and avoided the disaster. The missing thing was not a better confidence number. It was a representation of the model's own ignorance, which is to say a representation of basis. This is a third high-consequence domain, autonomous vehicles, joining the industrial safety system and the airliner in documenting the same failure.
Even the rigorous guarantee is the wrong shape
At the other end of the apparatus sits its most rigorous instrument. Conformal prediction wraps any trained model and produces not a single answer but a set of answers, guaranteed to contain the true one with a probability the user chooses, and it does so without assumptions about the data's distribution (Angelopoulos and Bates, 2021). It is the closest the field comes to a defensible, hard guarantee about a model's output. And yet its guarantee is of the wrong shape for this problem, by the method's own account. The coverage it promises is marginal: it holds on average over the distribution of cases, not for any particular case (Angelopoulos and Bates, 2021). A ninety percent guarantee means that across many predictions, about one in ten will be wrong, and the method does not, and cannot, tell you which ones. The guarantee is a statement about the long-run behaviour of the procedure, not a warrant that this claim, the one now in hand, is sound enough for the irreversible action about to be taken on it.
The whole apparatus answers a different question
Taken together, the ladder settles something the single calibration finding could not. The argument that confidence is the wrong category is not a complaint about immature tooling that better uncertainty quantification will one day fix. It survives the best the field has. A scalar confidence, a decomposition into noise and model-ignorance, and a distribution-free coverage guarantee are all, at bottom, statements about the estimation process and its behaviour across a distribution. None of them is a warrant that a specific claim's basis is sufficient for a specific action. The mismatch is not one of quality but of kind: between the distributional and procedural guarantees that machine learning knows how to produce, and the per-claim, action-relative admissibility that acting safely at a boundary requires.
This places uncertainty quantification exactly where the other near-neighbours landed. A conformal set, or an estimate of epistemic uncertainty, is precisely the sort of thing a representation of basis could carry and a downstream gate could weigh. Uncertainty quantification is a candidate input to admissibility, not a substitute for it. It can feed the model this work builds; it does not supply it.
References
Angelopoulos, A. N. and Bates, S. (2021). A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv preprint arXiv:2107.07511. arxiv.org/abs/2107.07511 · read copy
Kendall, A. and Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems 30. arxiv.org/abs/1703.04977 · read copy