Who do you call when the model is wrong?

Support processes were designed for software that fails loudly. Something crashes, a log records it, a ticket gets raised, an engineer traces it to a line of code. An SI system that is confidently wrong does none of that. It produces a plausible answer, nothing logs an error, nobody notices for three weeks, and when somebody finally does, there is no stack trace and no obvious owner. That gap is the single biggest unsolved problem in supporting these systems, and it is a process problem rather than a technical one.

Why the usual process does not fit

There is no error. The system did what it was asked. It was just wrong, in a way that required domain knowledge to notice.

The boundary is unclear. Was it the model, the prompt, the retrieval, the data, the configuration, or a change in the world that nobody told the system about? Five layers, often five different owners.

Reproduction is hard. The same input may not produce the same output, and the state that produced it — what was in the graph, which model version was loaded, what the policy allowed — has usually moved on.

Severity is subjective. “The forecast looked off” is not a severity level. Two people will grade it differently and both will be defensible, which means the first conversation is a negotiation instead of a diagnosis.

What has to exist before the incident

None of this is fixable at 3am. It has to be in place beforehand, and it is exactly what a support contract on an SI system should require.

Decision records. What the system was given, which model version, which policy applied, what it concluded and with what confidence. Without this, every investigation is archaeology. On the Kyber stack this is the trajectory every KyberCell keeps, as every step on the record describes.

Defined severity in business terms. Not “the model is wrong” but: a payment was prepared incorrectly, a customer received something they should not have, a report used in a decision was materially off. Written down, so the grade is a lookup rather than an argument.

A named owner per layer. Model, data, configuration, integration. It can be the same person, but the layers have to be enumerated so nobody discovers the gap during the incident.

A rollback path. What does “turn it off” mean here? Back to proposal-only, back to a previous model version, back to the manual process. Decided in advance, because deciding under pressure produces the wrong answer.

What we commit to

Characterisation within the response window. Not a fix — an answer to which layer this came from, which is what unblocks everybody else.

Reduced autonomy as the immediate action. The first response to a wrong decision is to narrow what the system may do unattended, then investigate. Cheap, reversible, and it stops the bleeding.

A written post-incident record. What happened, which layer, what changed, and what would catch it next time. Boring, and the only mechanism that makes these systems get better rather than accumulate folklore.

The uncomfortable part

Some of these incidents will be nobody’s defect. The model was within its accuracy, the configuration was right, the data was as described, and the outcome was still wrong — because that is what a probabilistic system does at the edges.

A supplier who promises this will never happen is lying, and a customer who expects it has misunderstood the technology. What can be committed to is that it is noticed quickly, characterised honestly, and bounded so that the cost of being wrong stays small.

That is what accountability for an SI system actually looks like. Not a guarantee of correctness — a guarantee that somebody is looking, and that the blast radius was decided in advance.