Where should retry live when your worker reads from a queue?

Where should retry live when your worker reads from a queue?

In two places, and you declare failure only when both have given up.

That sounds like belt and braces. It is not — the two tiers protect against different failures, and a system with only one of them is exposed in a way that is invisible until the day it matters.

For the engineering lead whose worker calls something that fails sometimes. 7 minute read.

Two places retry can live

Tier one is inside the process. The worker receives a message, calls the model, gets a failure, and tries again — a small number of attempts with exponential backoff, all within the same execution, on the same message it is already holding.

Tier two is the queue itself. If the worker never deletes the message, the visibility timeout expires and the queue hands it to a consumer again. The message is redelivered a limited number of times before it is considered undeliverable.

Our contract-review pipeline runs both: three in-process attempts with exponential backoff, beneath three deliveries from the queue. A message is marked permanently failed only when both tiers are fully exhausted.

These look like the same mechanism at different scales. They are not. They recover from different classes of failure, and that is the whole argument.

Why either one alone is wrong

In-process retry alone dies with the process. A transient model error is handled beautifully — until the container is terminated by a deploy, an autoscaler or a crash. The retry state lived in memory, the message was never deleted, and whether that work survives depends entirely on the queue. If you are relying on in-process retry as your only mechanism, you are relying on your process never dying, which is not a property containers have.

Queue redelivery alone is wasteful and slow. A model call that fails on a network blip could have succeeded a second later. Instead the worker gives up, the message becomes invisible for the length of the visibility timeout, and the retry happens tens of seconds later on a fresh consumer that has to re-establish everything it already had — provider clients, configuration, any per-job cache.

So the split is not redundancy. It is specialisation. Tier one handles failures that are transient in time. Tier two handles failures that are transient in place — the process, the container, the host.

Failure is a conjunction, not a disjunction

The rule worth stating precisely: a message is only permanently failed when in-process retries are exhausted and the queue's redeliveries are exhausted.

That is a stricter condition than most systems implement, and it has a pleasant consequence — the easy failure modes are absorbed silently. A blip inside a message never surfaces. A container death mid-message never surfaces. Something has to be genuinely, repeatedly broken to reach the dead-letter queue.

Which means your dead-letter queue becomes meaningful. If everything transient is absorbed underneath, arrival in the DLQ is a real signal rather than background noise — and a DLQ nobody trusts is a DLQ nobody reads.

The multiplication nobody budgets for

Here is the part that surprises people, and it is arithmetic rather than architecture.

The tiers multiply. They do not add. Three in-process attempts beneath three redeliveries is up to nine attempts at the same unit of work — not six.

For a queue worker calling a database, nine attempts is nothing. For a worker whose retried operation is a model call, nine attempts is nine model calls, and you are paying for every one of the failed ones. On a pipeline where a single document produces hundreds of chunk messages, a provider degradation does not cost you a little more. It costs you a multiple.

And the latency compounds worse than the cost. Within a tier, exponential backoff means each attempt waits longer than the last. Between tiers, the wait is the visibility timeout — typically tens of seconds, and deliberately longer than the work takes. A message that exhausts everything has spent minutes failing.

The design lesson: choose your tier depths against what the retried operation costs, not by copying defaults. Three-by-three is reasonable for cheap idempotent work. For expensive model calls, fewer in-process attempts and more reliance on redelivery is often the better trade — because redelivery is free and the model call is not.

What both tiers demand of your worker

Neither tier is safe unless the worker is idempotent, and tier two makes that unavoidable: the message will be processed more than once, by different processes, possibly concurrently.

Our pipeline handles this with an atomic conditional update that claims a chunk before work begins — two workers cannot both own it, and a redelivered message finds the claim already made. Our published piece on queue topology covers the same requirement from the infrastructure side, keyed on chunk and agent identity.

The requirement is easy to state and easy to get wrong, because a worker is usually idempotent for its main write and not for its side effects. If processing also increments a counter, emits a metric or appends to a log that something else reads, those run again on every retry — and the counter is now wrong in a way nothing will flag.

Where it breaks

The circuit breaker changes the arithmetic and it is easy to miss. If a breaker is protecting the model and it is open, tier-one retries do not happen at all — they fail immediately by design. Your three in-process attempts become zero, and the message falls straight through to tier two.

That is correct behaviour and it makes the tiers behave differently under load than in testing. In a healthy system you get nine attempts; during a provider outage you get three, spread across redeliveries, arriving much faster than you would expect. Anyone tuning retry depth against normal conditions is tuning against the case that matters least. Chapter 4 covers the breaker itself.

And a poison message costs the full budget. Malformed input that will never succeed still consumes every attempt in both tiers before it lands in the DLQ. Retry cannot distinguish "the model is temporarily unavailable" from "this input is permanently unprocessable", and it treats both the same. Validation before the retry loop is cheaper than nine attempts at something that was never going to work.

What we would build, and the check for Monday

How we build it. Set tier depths from the cost of the retried operation rather than from a template, and make them configuration rather than constants so they can be tuned without a deploy. Then record which tier resolved each success. A system that recovers mostly in tier one has a flaky dependency; a system that recovers mostly in tier two has flaky infrastructure, and those are different problems with different owners.

Status, stated plainly. This is a system we built. It runs, it is demonstrable on a call, and it has zero delivered engagements. We build these on request, against your systems. No number here describes anyone's business but our own.

The check worth running, and it needs nothing from us. Multiply your in-process attempt limit by your queue's redelivery limit. That product is how many times your most expensive operation can run for one unit of work. If you have never seen that number before, you have not costed your failure path — and it is the number that arrives on your bill during an outage, not during a normal week.

About Cognilium Cognilium is an AI engineering company — agent systems, retrieval, knowledge graphs, voice AI and the production plumbing that makes them survive contact with real workloads. https://cognilium.ai · https://www.linkedin.com/company/37180269/

Want to know what your failure path costs? Book a 15-minute call — we will walk the arithmetic with you, on your queue configuration if you bring it. No deck.

Sources

This article draws on our own build. Dead-letter configuration per queue stage is covered in our published piece on mixing FIFO and standard queues in an agent pipeline, and is cited rather than repeated here.