Skip to content
Paula Livingstone writing · projects · tools

Projects

Attestable

When probabilistic AI feeds safety-critical systems, unproven output can drive physical action unchecked. Attestable is a discipline for marking a claim's basis, so it cannot.

OT SecurityCyber-Physical AssuranceIEC 62443ProvenanceSafety CaseLLM Agents

When work is handed forward, to the next session, the next shift, the next engineer, what arrives is a body of state: findings, conclusions, current values, things believed to be true. Almost none of it arrives marked. The measured sits alongside the assumed, the verified alongside the inferred, in the same voice and the same typeface. The next owner picks it up and acts on it, and in doing so silently promotes every assertion in it to a fact. Attestable is a discipline for refusing that promotion: for marking which parts of carried-forward state are proven, by what, and therefore safe to act on.

That human handover is the accessible version of the problem, and a good way in, but it is the gentle case. The boundary this work is really about sits deeper, inside cyber-physical systems, at the interface where a probabilistic subsystem hands state to a deterministic one. On one side, a model, an estimator, an inference engine, something that produces values by sampling a distribution. On the other, a valve, a relay, an interlock, a controller, something that consumes a value and acts on it mechanically. Between them there is no shift meeting, no colleague to ask a question of, no shared language in which doubt could be voiced. The value crosses, and the mechanism moves. Everything that makes the human case recoverable is absent, and what is left is the promotion in its purest and most dangerous form.

The problem it names: False Determinism

The failure has a name: False Determinism, the moment a claim acquires the authority of settled fact not because it was proven, but because it was carried across a boundary and nobody marked it otherwise. A probabilistic system does not distinguish, in its output, between what it has established and what it has merely produced. Both come out fluent. Both come out in the register of assertion. A model that has measured a value and a model that has estimated one will hand you both in the same declarative sentence, and the sentence carries no signal about which is which. This is not a defect to be trained away; it is what generation is. The consequence is that the burden of telling proven from asserted falls entirely on the receiver, at exactly the moment the receiver is least equipped to carry it, arriving cold, without the working, with only the conclusion in hand.

The failure is quiet, which is what makes it dangerous. Nothing announces itself. The unverified claim does not look unverified. It looks like the rest of the handover, and it is acted upon like the rest of the handover, and the error it seeds surfaces later, downstream, somewhere the connection back to its origin has already been lost.

Everything in that description is threat-shaped, and the instinct it summons is to hunt: a quiet signal, indistinguishable from the legitimate, surfacing as harm downstream, that is the silhouette of a stealthy intrusion, and the trained response is detection. The instinct is wrong here, and following it misplaces the whole problem. There is nothing concealed in a falsely deterministic value, and no one concealing it. Pull it apart to any depth and there is no tell, no residue of origin, nothing hidden to be found, because nothing was hidden. What is wrong with the value is not something it contains but something it never carried: no record of how it came to be. And there is no deceiver to unmask, because the thing that produced it held no stance toward its truth at all. A lie inverts the truth and a mistake misses it, but both are relationships to it; a generated value has none. It was never in the business of being true or false, and was promoted into a setting that can read it only as one or the other.

This borders territory that philosophy has already named. Frankfurt's account of assertion made "with no regard for the truth," recently turned on language models directly by Hicks, Humphries and Slater, describes the same indifference at the level of the speech act. The concern here is downstream of that and more mechanical: not what kind of utterance this is, but what happens when a value with no relation to its own truth reaches a consumer that can only treat it as settled. That consumer is the boundary the discipline has to guard, and it is why the remedy is not detection. There is nothing to detect. The answer is not a sharper eye on the traffic but an interface that can represent what the artefact cannot: whether, and on what basis, the value was ever proven.

The two names do different work, and the distinction is the whole point. False Determinism is the diagnosis, it describes what goes wrong, and it would go wrong whether or not anyone had a remedy. Attestable is the discipline built in response: the practice that makes the false promotion something a system can refuse rather than something it absorbs unnoticed.

This collapse of surface signal has an exact precedent in network security, and the precedent is instructive because its remedy is known. Firewalls once discriminated traffic largely by port: the port number was a cheap attribute, present in the packet, that correlated with the nature of the traffic behind it. That correlation collapsed when applications converged on port 443. Once everything arrived over the same port, the inspected attribute no longer discriminated anything, and inspection had to move up the stack, to attributes that were not present in the packet and had to be established by other means. The evidential surface of a claim has undergone the same collapse. Register, hedging, and provenance-by-style were weak signals, present in the artefact, of how far a claim had been established; fluent generation has driven every claim into the same declarative envelope, and those signals no longer separate the measured from the merely produced. The lesson transfers with the failure: when the discriminating signal is no longer recoverable from the artefact, it has to be attached to it instead. Which is the subject of what follows.

What an attestation is

An attestation is an artefact that rides alongside a piece of carried-forward state and says how far that piece has been proven and on what basis. Not a confidence score. A confidence score is the wrong kind of information for the job, however honest and however well calibrated, it is a statement about the generator's own internal state, not evidence that the thing asserted is so. It tells you how sure the system is; it does not tell you what the system did to earn that surety, and it is the latter the next owner needs. An attestation is closer to a provenance record: this value was measured, by this means, at this time; this conclusion was inferred from those two measurements; this item is a model assertion and has been tested against nothing.

The vocabulary is deliberately narrow. Attestable is the discipline. To attest is the act of marking a piece of state with the basis on which it is held. An attestation is the artefact produced by that act, travelling with the state it describes. A given claim is attested or unattested, and the second of those is the important one, because the whole point is that unattested is a legible, sayable status rather than an absence nobody notices.

Why the next owner is the unit that matters

The framing is deliberately about handover rather than about truth in general. A claim does not need to be proven in the abstract; it needs to be proven enough for the specific thing the next owner is about to do with it. That is a lower bar in some places and a far higher one in others, and it can only be set with reference to the action downstream. An estimate that is perfectly adequate for deciding where to look next is not adequate for deciding whether it is safe to energise. Same claim, same confidence, different admissibility, because the next owner is doing something different with it.

So attestation is not a global grading of a statement's truthfulness. It is a statement about fitness for the next action, which means it is only meaningful when the next owner and their next action are actually in view. Attestable takes that seriously: the question is never just is this true, it is is this proven to the standard that the person about to act on it requires.

Where the boundary becomes physical

Return to the interface between the probabilistic and the deterministic subsystem, because this is the centre of the problem, not an extreme case of it. When the next owner is a valve or an interlock, acting on an unproven claim does something to the world that cannot be undone by correcting the record. And the deterministic side cannot defend itself, because it has no way to receive the claim as anything other than settled. Its input interface accepts a value. It has no field for this is only inferred, no type for unattested. A value that arrives with no basis is byte-for-byte indistinguishable from one that was proven, so the mechanism actuates on either without ever knowing the difference. False Determinism here is not a mistake a careless operator makes; it is the default behaviour of an interface that cannot represent the alternative.

This is why the discipline has to sit on the boundary itself. Attestation is not something the actuator can be asked to check, because a valve does not check anything. The admissibility decision has to be interposed before the value reaches the deterministic subsystem: a gate on the boundary that reads the attestation, weighs it against what the impending action requires, and either passes the value through or refuses it. Where an unattested value would otherwise have silently actuated a mechanism, the gate is what holds the status the actuator's own schema cannot, and refuses by default. That placement, on the boundary, adjudicating fitness for the next action at the point where a probabilistic value would otherwise become a physical act, is the move that distinguishes this from merely recording where a claim came from.

None of this machinery is new to safety-critical engineering. Operational technology and cyber-physical assurance have always closed the gap between believed safe and demonstrated safe with exactly this discipline: nothing is treated as proven because it was asserted by someone competent and confident; it is proven because a named party attested to it, on a stated basis, and put their name to the attestation. That machinery is old, and it works. Attestable is the observation that probabilistic subsystems have just introduced, at enormous scale, precisely the failure mode that machinery was built to prevent, and that the discipline built to catch it transfers, once it is placed where the probabilistic meets the deterministic.

An ongoing practice

The substance of the work is the taxonomy: the kinds of carried-forward state, and what counts as an admissible proof for each. That is being written up, and it is the part that gives the discipline teeth, without it, Attestable is a good instinct rather than a method. This page is the standing description of the project; the taxonomy and the individual pieces will be written up under it as they are settled.