What stops a degraded model from taking the whole queue with it?

What stops a degraded model from taking the whole queue with it?

A circuit breaker. And in a container fleet it protects considerably less than most teams assume, because every replica has its own.

The mechanism is worth understanding properly, because the failure it prevents is one your retry logic actively creates.

For the engineering lead whose pipeline calls a model provider that is occasionally unwell. 7 minute read.

Retry is the amplifier

Start with what goes wrong without one.

A model provider degrades — not down, just slow and erroring intermittently. Your workers call it, fail, and retry. Retry means the provider that is already struggling now receives more traffic than when it was healthy, because every failure produces additional attempts.

Meanwhile each worker is occupied for the full duration of its retry sequence: attempt, backoff, attempt, longer backoff, attempt. Workers that would normally clear a message in seconds are now held for the length of a failing sequence, so throughput collapses while the queue keeps filling.

Neither effect is a bug. Both are retry working exactly as designed, and together they convert a degraded dependency into a stalled pipeline. That is the failure a breaker exists to interrupt.

The state machine

Three states, and the transitions are the whole design.

CLOSED is normal: calls pass through, failures are counted.

OPEN is reached after a few consecutive failures. Calls no longer reach the provider at all — they fail immediately. The provider stops receiving traffic from this worker entirely.

HALF_OPEN arrives after a recovery interval. One call is allowed through as a probe. Success closes the breaker; failure opens it again for another interval.

Our pipeline opens after three consecutive failures and waits one hundred and twenty seconds before probing. Those are our numbers for our workload, and §5 argues about what they cost.

The important property: OPEN is not an error state, it is a decision. The system has concluded that calling the provider right now is not worth the attempt, and acting on that belief immediately is cheaper than rediscovering it on every message.

Failing fast is the kind option

There is a version of this that sounds callous — the system gives up rather than trying — and it is worth arguing the other way.

A breaker is better for the failing provider, not just for you. A degraded service recovers faster without a retrying client hammering it. Backing off entirely for a couple of minutes is materially more helpful than continuing to send load that will fail.

And it is better for the work. A message that fails instantly is returned to the queue quickly and cheaply, ready to be tried later when conditions have changed. A message that fails slowly, after a full retry sequence, has consumed a worker for the whole duration and arrives at the same outcome.

Fast failure is not giving up. It is declining to spend money on an attempt you have good reason to believe will not work.

Every replica has its own breaker

Now the part that changes how much you should trust this, and it is rarely stated.

The breaker is an object inside a process. It counts failures seen by that process. Run ten containers and you have ten independent breakers, each with its own counter and its own view of the provider's health.

Two consequences follow, and the second is worse than the first.

The provider sees your failure threshold multiplied by your replica count. With ten replicas and a three-failure threshold, up to thirty failed calls can reach a struggling provider before the last breaker opens. The protection you configured is per-worker; the load the provider experiences is fleet-wide.

And a breaker that has opened protects nobody else. Worker four learning the provider is unwell does not tell workers one through three, who each have to discover it independently — by failing. Every replica pays the full tuition.

None of this makes local breakers wrong. They are simple, they need no coordination, and they cannot themselves become a shared point of failure — which is exactly why a shared breaker, in a store every worker consults, trades one problem for another. But the arithmetic should be deliberate: your effective threshold is your configured threshold times your replica count, and if you have not multiplied those two numbers you do not know what your breaker actually does.

The thresholds are a trade, not a default

A low failure threshold opens quickly and is prone to false positives. Three consecutive failures during normal operation is not rare on a busy provider; open on it and you stop working for two minutes over something that would have resolved on the next call.

A high threshold is slow to protect. By the time it opens, the amplification in §1 has already happened.

A short recovery interval probes often, discovering recovery quickly and adding load to something possibly still unwell. A long one is patient and leaves capacity idle after the provider is fine again.

There is no correct pair, only a pair matched to how your provider actually fails. A provider that fails in short spikes wants a fast probe. One that fails in long outages wants patience. Both are configuration in our pipeline rather than constants, which is the only defensible answer when the right value depends on a third party's behaviour.

Where it breaks

Half-open is a synchronised event across the fleet. Ten containers that opened at roughly the same moment reach the probe at roughly the same moment — so the "one careful probe" is ten simultaneous calls to a service that may still be fragile. The recovery mechanism can re-break the thing it is testing. Jitter on the recovery interval costs nothing and is routinely omitted.

A closed breaker means nothing was persistently wrong. It does not mean nothing was wrong. Failures below the threshold are absorbed silently by retry, so a provider degrading steadily without ever producing three consecutive failures in one worker looks entirely healthy to the breaker while quietly costing you real money in retries. The breaker is a circuit protector, not a monitor, and treating its state as your health signal will miss the slow degradation entirely.

And it interacts with partial-failure thresholds in a way worth checking. If a scoring pass tolerates some agents failing, an open breaker can make several fail at once — and the pass may still "succeed" on its minimum while having produced a materially thinner result. Chapter 5 is about that tolerance and chapter 1 about what it does to routing.

What we would build, and the check for Monday

How we build it. Keep breakers local, add jitter to the recovery interval, and export the state as a metric rather than reading it as one. Then track the failures the breaker never saw — the ones absorbed by retry beneath the threshold — because that is where a slow degradation lives and no breaker will tell you about it.

Status, stated plainly. This is a system we built. It runs, it is demonstrable on a call, and it has zero delivered engagements. We build these on request, against your systems. No number here describes anyone's business but our own.

The check worth running, and it needs nothing from us. Multiply your breaker's failure threshold by your replica count. That is how many failed calls your provider can receive before your fleet stops sending — and it is almost certainly larger than the number you thought you configured.

About Cognilium Cognilium is an AI engineering company — agent systems, retrieval, knowledge graphs, voice AI and the production plumbing that makes them survive contact with real workloads. https://cognilium.ai · https://www.linkedin.com/company/37180269/

Want to know what your breaker actually protects? Book a 15-minute call — we will walk the arithmetic with you, on your configuration if you bring it. No deck.

Sources

This article draws on our own build. Retry mechanics are covered in the sibling chapter on where retry should live, and partial-failure tolerance in the chapter on thresholds.