Skip to content
Paula Livingstone writing · projects · tools

Attestable Evaluation

Failure Analysis: Two Failures of Different Kinds

The two mismatches are not variants of one problem. They occur at different layers and have different remedies, and treating them together would obscure both.

An observability failure at the construction boundary

A spreadsheet applied a fixed formula to a figure a model had produced, and presented the result as a calculation. The calculation was correct. Nothing presented with it recorded that the input had been generated, so the record carried only the calculation's own origin. The demand excluded generated ancestry, found none, and admitted.

The failure location matters and must be stated precisely. This is an unsafe admission by the end-to-end artefact under the tested ingestion conditions. It is not evidence that the kernel evaluated visible ancestry incorrectly: the kernel behaved consistently on the record it received, and the ancestry was absent from that record because the derivation happened outside anything the discipline mediated.

Origin conservation cannot conserve an event that never entered the record. The design chapter states this limit, and the case was authored to test whether the limit is reported honestly rather than concealed. It is reported honestly, by failing.

model estimategeneration event spreadsheet appliesa fixed formulano record of the step record presentedorigins: the calculation only demand admitsno generated origin found outside the discipline the generation is never unioned into any origin set origin conservation conserves what crossed the boundary, and nothing else
Figure 8. Where the information is lost in S-ancestry-1. The dashed region is outside anything the discipline mediates: a person applies a formula to a model's estimate and enters the result, and no system records that the input was generated. The record that reaches the boundary represents everything the construction process supplied, the kernel evaluates it correctly, and the admission is unwarranted. No demand-language construct could recover the missing event, which is why this failure is classified at the construction boundary rather than in the demand language.

The practical consequence is serious for a library intended to receive claims from existing plant tooling. An engineer who reads a model's estimate on a screen, performs the arithmetic by hand, and enters the result as their own figure defeats the mechanism completely, and no record anywhere shows it. The strength of origin conservation is bounded by construction-boundary coverage and by the truthfulness of what crosses it, neither of which is a property of the formal model.

It also bears on any future work. No inference operator could recover an event that never entered the record, so treating this as a demand-language deficiency would misdiagnose it.

A translation-validation failure

The paired case describes an act whose author is entitled to decide having seen a generated ancestry: the requirement says explicitly that the act may proceed or not on that judgement, the point being that what the figure rests on is visible rather than concealed by the calculation. It was translated with a demand excluding generated ancestry outright, which encodes a different act.

The contradiction was present in the sealed translation before anything ran. The case recorded an expected verdict of admission beside a predicted decisive result of failure on the ancestry axis, and a verdict cannot be an admission while its decisive axis fails.

The checker did not test for this. Execution prompted inspection of a contradiction already present in the sealed translation, exposing a missing validation invariant. The invariant is elementary and mechanical: an expected admission requires every predicted decisive result to pass, an expected refusal requires at least one named failing axis, and an expected undetermined requires at least one unknown axis and none failing.

The consequence is twofold. For discrimination, the mismatch is a translation failure rather than an artefact failure: the artefact did what the demand asked. For representation, the case was not exact at all but a conservative overconstraint, which is why the validity-adjusted count is four rather than five. The frozen corpus contained one case that its own translation rules should have excluded from execution.

That is a methodological result of some value, and it should not be overstated. The protocol did not detect the defect automatically; a human inspection prompted by an unexpected verdict did. What the protocol supplied was a sealed record in which the contradiction was visible and attributable once someone looked.