The problem statement rests on a premise it has so far only asserted: that the next owner of a claim relies on it, acts on it, and in doing so promotes it to a fact. That premise is not obvious, and it is not free. It is a claim about how people, and systems, behave when they receive conveyed state from something that produced it. There is a mature field that has studied exactly this for forty years, under the heading of trust in automation, and it grounds the premise firmly. It also completes a thread left hanging earlier: when the safety-assurance literature named automation bias and handed it onward, this is the field it handed it to.
The framework: reliance without checking
The foundational work in this field gives a taxonomy of how people relate to automation: they use it, misuse it, disuse it, or abuse it. The term that matters here is misuse, defined as overreliance on automation, resulting in failures to monitor it effectively (Parasuraman and Riley, 1997). That is precisely the failure this work is about, stated in the vocabulary of human factors: a person receives the output of an automated system and does not adequately check it before acting. The structure is the same as a handover. At higher levels of automation, the field notes, the machine carries out a function and informs the operator of the result, but the operator cannot see into how the result was reached (Parasuraman and Riley, 1997). The operator receives a conveyed value and acts on it, with no view of its basis.
The same failure, with bodies, in a second domain
This is not a theoretical worry. The foundational review catalogues accidents that are this thesis made physical, in aviation rather than in a refinery. In a class of controlled-flight-into-terrain accidents, the crew selected the wrong guidance mode, and, in the review's own words, the indications presented to the crew appeared similar to when the system was tracking the glide slope perfectly. The automation conveyed a state that was indistinguishable, on its surface, from the correct one. The crew acted on it, and the aircraft flew into the ground. In another case a pilot with low confidence in his own manual skills relied heavily on the autopilot, failed to monitor the aircraft's airspeed, and crashed. The conveyed state carried no visible mark of how far it could be trusted, and it was promoted to fact at the controls.
Two independent, high-consequence domains, industrial safety systems and civil aviation, document the same failure. That makes it general rather than a quirk of any one setting.
What the field concluded, and where it stops
Carried into the age of artificial intelligence, this work supplies the terms the argument needs. The over-trust it describes has a name, automation bias, the tendency to accept an automated system's output even in cases where it is clearly incorrect. A sharper phrase from the early literature is blind reliance: acceptance of a machine's actions without question of its intent or motives. A receiver promoting conveyed state to fact without checking its basis is not a novel diagnosis; it is a documented human tendency with a fifty-year research record.
And the field's own remedy points exactly where this work goes. The goal it converged on is appropriate trust, or trust calibration: reliance that matches the actual trustworthiness of what is relied upon (Mehrotra et al., 2023). Crucially, calibrating trust well is understood to require insight into the automation's purpose, its process, and its performance, which is to say its basis, not merely its output (Lee and See, 2004). The human-factors field has spent decades establishing that reliance should be calibrated to basis. But its remedy is always addressed to a human operator, asked to calibrate better, trained to monitor more carefully. That remedy has no purchase at a boundary where the receiver is not a human who can be trained but a mechanism that actuates. The field diagnosed the need to calibrate reliance to basis, and left the basis to be judged by a person. This work builds the mechanism that carries the basis across the boundary, so that the judgement can be made where no person is present to make it.
References
Parasuraman, R. and Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors 39(2). doi.org/10.1518/001872097778543886 · read copy
Mehrotra, S., Degachi, C., Vereschak, O., Jonker, C. M. and Tielman, M. L. (2023). A Systematic Review on Fostering Appropriate Trust in Human-AI Interaction. arXiv:2311.06305. arxiv.org/abs/2311.06305 · read copy
Lee, J. D. and See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors 46(1). doi.org/10.1518/hfes.46.1.50_30392