Is partial failure an outage, or a setting?

Is partial failure an outage, or a setting?

Twelve agents score a chunk of text. Three of them fail. Has the job failed?

There is no universally right answer, which is exactly why it should be a decision you made on purpose rather than one your exception handling made for you.

For the engineering lead deciding what "done" means when work is spread across many model calls. 7 minute read.

Three of twelve just failed

At any real scale, some model calls fail. Not because the system is broken — because you are making a great many of them, against a service you do not control, over a network.

A pipeline that treats every failed call as a failed job will fail constantly. One chunk, twelve scoring calls; a contract, dozens of chunks. Multiply that out and the probability of every call succeeding is not close to one.

So "all or nothing" is not a conservative choice. It is a choice to fail often. The alternative is to decide, in advance, how much of the work has to succeed for the result to be worth having.

Two thresholds, not one exception

Our contract-review pipeline makes that explicit at both stages where fan-out happens.

Scoring succeeds if at least a configured minimum of the twelve agents return. Not all twelve. A number you set.

Analysis succeeds if at least a configured ratio of the selected analysts return. A ratio rather than a count, because the number of analysts selected varies per chunk — routing might have chosen one, or all of them.

Both are configuration, not constants. That distinction is the entire argument: the acceptable degree of incompleteness is a property of what the output is for, and that changes between customers, between document types and between a first pass and a final review.

Note what this is not. It is not a try/except that swallows errors. The failures are counted, the threshold is evaluated deliberately, and falling below it fails the unit properly.

Degradation as a designed state

The shift worth naming: most systems have two states, working and broken. This one has three.

Working. Degraded but acceptable. Broken.

That middle state is where a distributed system actually spends its unhappy time, and having no name for it is why so many pipelines are brittle.

Without it, a single failed call among dozens forces a binary choice: pretend nothing happened, or throw everything away. Both are wrong, and teams usually pick the first by accident — an exception handler that logs and continues is an undeclared threshold of zero.

Making it explicit turns an accident into a policy. The system is allowed to be partly successful, it knows when it is, and somebody chose the boundary.

What the threshold actually trades

A high threshold buys completeness and spends availability. Require eleven of twelve and you will fail jobs that a human would have found perfectly useful.

A low threshold buys availability and spends completeness. Require four of twelve and almost nothing fails — and some results are thin in ways nobody downstream can see.

The right value depends on what happens next, and specifically on who checks. A result that a specialist will review carefully can tolerate more incompleteness than one that feeds an automated decision. The threshold is really a statement about how much you trust the reader to notice something missing — which makes it a product decision that happens to be expressed as a number in a config file.

The result carries no denominator

Here is the problem this creates, and it is the reason §7 exists.

A result produced by all twelve scorers and a result produced by the minimum acceptable number look exactly the same. Same shape, same fields, same confidence in the interface. The output format has no place to say "three of the categories were never assessed."

That matters more than it first appears, because of what a missing score does downstream. In a routed system, a category that scored as absent looks identical to a category that scored as not relevant — so a chunk can be under-routed by a failure rather than by a judgement, and the finding that was never produced is indistinguishable from the finding correctly judged unnecessary.

The system knows the difference at the moment it happens and then discards it. The count is right there in the scoring phase; unless something records it, it is gone by the time anyone looks at output.

This is the real cost of partial-failure tolerance, and it is not the incompleteness. It is that incompleteness becomes invisible. A brittle system that fails loudly at least tells you something went wrong.

Where it breaks

A threshold can mask systematic failure indefinitely. If one particular agent fails consistently — a malformed prompt, a schema the model keeps violating, a category nobody maintained — the job still succeeds every time, because eleven others cleared the bar. You have configured a system that will never tell you one of its twelve components has been dead for a month.

And a threshold nobody has ever breached teaches you nothing. Either it is well chosen or it is never tested, and from the outside those look identical. The first time it is breached is usually the first time anyone finds out what the system does next.

Nor do thresholds compose the way people expect. Scoring can pass its minimum and analysis can pass its ratio, and the combined result still be materially thinner than either implies — because the failures were not independent.

A degraded provider produces correlated failures, which is precisely when a circuit breaker is opening and removing retry attempts. The thresholds were chosen against independent random failure and are being asked to hold during correlated failure, which is the case they are worst at.

What we would build, and the check for Monday

How we build it. Record the denominator with the result — how many agents were asked, how many returned — and carry it through to anything that consumes the output. A downstream reader deserves to know whether it is looking at a complete assessment or an acceptable one.

Then alert on the composition of failures rather than the rate. Ten failures spread across ten agents is weather. Ten failures from the same agent is a broken component hiding behind a threshold that is doing its job.

Status, stated plainly. This is a system we built. It runs, it is demonstrable on a call, and it has zero delivered engagements. We build these on request, against your systems. No number here describes anyone's business but our own.

The check worth running, and it needs nothing from us. Find the place in your pipeline where a partial result becomes a whole one. Then ask whether anything downstream can tell the difference. If your output has no field for how complete it is, every consumer is treating your degraded results as authoritative ones — and they have no way to know they should not.

About Cognilium Cognilium is an AI engineering company — agent systems, retrieval, knowledge graphs, voice AI and the production plumbing that makes them survive contact with real workloads. https://cognilium.ai · https://www.linkedin.com/company/37180269/

Want to know what your pipeline is quietly calling a success? Book a 15-minute call — we will walk the thresholds with you, on your configuration if you bring it. No deck.

Sources

This article draws on our own build. The routing that consumes these scores is covered in the sibling chapter on scorers and analysts; the circuit breaker referenced in §6 has its own chapter.