Your Retry Just Sent the Email Twice

A single model call fails all at once, so you retry it and move on. A multi-agent run fails in the middle, after it has already sent the email, and the retry sends it again.

The last chapter was about an attacker reaching the tool: a poisoned document riding the hand-offs until it reached the agent that could send email, and making it fire on the wrong instruction. This chapter is the same tool boundary with no attacker in sight. Nothing was poisoned. The pipeline did exactly what you built it to do, and it still went wrong, because halfway through the run a model call timed out, a retry kicked in, and the send-email tool that had already fired once fired a second time. The security chapter worried about the tool doing the wrong thing. This one is about the tool doing the right thing twice.

Here is the uncomfortable part. The reflex that keeps a single-call system alive is a retry with backoff, and it is correct there, because a single model call fails atomically: it either returns or it does not, there is no half-finished state, and retrying the whole thing is safe. That reflex is not just useless in a multi-agent system, it is the bug. A multi-agent run does not fail atomically. It fails in the middle, after some agents have already committed real, irreversible actions, and a retry that replays the run replays those actions. The thing you added to make the system reliable is the thing that duplicates the side effect.

TL;DR

A single model call fails atomically, so retrying the whole call is safe: there is no partial state to corrupt. A multi-agent run is a sequence of model and tool calls with no transaction around it, so it fails in the middle, after some agents have already committed real side effects like a sent email, a written row, or a submitted filing, and there is no rollback. This inverts the usual fix. A blind retry replays the steps that already succeeded and fires their side effects again, so the retry is not recovery, it is a second bug. Reliability in a multi-agent system is structural, at the action boundary, not a wrapper around the run. Make every side-effecting tool idempotent by giving it an idempotency key derived from the run and the logical action, so a replay returns the first result instead of acting again. Checkpoint each completed step to durable storage so a retry resumes at the failed step instead of re-running the whole pipeline and paying every token twice. For the steps you cannot make idempotent, a third-party charge or an irreversible external submit, define a compensating action, the saga pattern, because there is no rollback, only a forward action that cancels a prior one. And classify failures before retrying, because a timeout is transient and worth retrying while a schema violation is deterministic and will fail identically every time, so retrying it is a slower way to fail. The failure rate of any one call is small. In a run of fourteen calls it compounds into roughly one run in eight landing in a partial state, and what decides your reliability is not whether that happens but what a retry does when it does.

A single call fails all at once. A multi-agent run fails in the middle.

In a single model call the failure model is simple. The call returns a result or it raises an error, and there is no third state. Nothing happened halfway. So the recovery is the simplest thing in distributed systems: retry the whole call, maybe with backoff, and you are done. You can retry it five times and the worst case is five identical attempts, because a call that failed did not leave anything behind.

A multi-agent run has no such property, and the reason is that there is no transaction wrapping it. Take the same six-agent extraction pipeline this series has used throughout: a planner, a retriever, two extraction workers, a reconciler, and a reviewer that holds the two tools that touch the outside world, finalize to the system of record and send email. A run walks through those agents in sequence, and every one of them is a separate model call, several of them making separate tool calls. There is no COMMIT at the end and no ROLLBACK on failure, because the run is not a database transaction. It is fourteen independent calls in a trench coat. When the reviewer sends the client email and then the finalize call times out, the run has failed, but the email is already gone. Half the side effects are real and permanent. The other half never happened. The run is not failed or succeeded. It is stuck exactly halfway, and nothing in your stack knows how to feel about that.

The retry is the trap

The instinct is to wrap the run in the same retry you would wrap a single call in. A run failed, so run it again. In a stateless call that is exactly right. Here it replays the planner, the retriever, both extraction workers, the reconciler, and then the reviewer, and the reviewer sends the email a second time. You did not recover the run. You duplicated its side effect. The client now has two identical emails, or two filings, or two rows in a system of record that a downstream process will read as two separate events.

It is worse than a single accidental double, because retries hide in every layer of the stack and none of them coordinate. The model SDK retries on a 429. The job queue redelivers a message it did not see acknowledged. The load balancer times out and the client library retries the request. The user, watching a spinner that never resolves, hits the button again. Each of these is a reasonable local decision, and stacked on a run that takes irreversible actions, each one is another copy of the email. The naive answer, turn off retries, is not an option either, because transient failures are real and a run that never retries anything simply fails more. You cannot remove the retry. You have to make the thing it replays safe to replay.

Make the side effect idempotent

The fix lives at the same action boundary the security chapter put guardrails on, and it is the same shape of fix: deterministic code at the tool, not a model's judgment. Every side-effecting tool takes an idempotency key, a string derived from the run and the specific logical action it is performing, and the tool deduplicates on that key. The send-email tool is called with a key of, say, a hash of the run identifier, the recipient, and the message template. The first call with that key sends the email and records the key with its result. The second call with the same key does not send anything. It finds the recorded result and returns it. The retry still happens, the reviewer still calls send-email, and the client still receives exactly one email.

This is the single highest-leverage move in the chapter, because it converts a retry from a liability back into the safe primitive it is supposed to be. Once every side effect is idempotent, you can retry the whole run as many times as you like and the outside world sees each action exactly once. A run retried three times sends one email, not three. The key has to be derived from the logical action, not generated fresh per call, or you have defeated the entire mechanism: a new random key every attempt is the same as no key at all. And the dedup has to live server-side, at or behind the tool, because a check the agent performs in its own reasoning is exactly the check a replay will perform again and pass again.

Checkpoint the run so a retry resumes, not restarts

Idempotency makes a replay safe. It does not make it free. If the run fails at the reviewer, the sixth agent, a blind retry still re-runs the planner, the retriever, both extraction workers, and the reconciler before it gets back to the point of failure, and every one of those is a fresh model call you pay for again. The observability chapter measured a single run of this pipeline at around 41,000 tokens. Re-running five of its six agents to recover from a failure in the sixth pays roughly five sixths of that bill, tens of thousands of tokens, to redo work that already succeeded.

The fix is to checkpoint. As each agent completes, write its output to durable storage keyed by the run identifier and the step. On a retry, before running a step, check whether its checkpoint already exists, and if it does, load the output and skip straight to the next step. The retry resumes at the reviewer instead of restarting at the planner. It pays for the one failed step and its tail, a few thousand tokens, instead of re-running the whole pipeline. The checkpointed outputs are the same notes the agents pass each other from the hand-offs chapter, with one addition: they are written down where a second attempt can find them, so the second attempt inherits the first one's work instead of repeating it. Idempotency keeps a retry correct. Checkpointing keeps it cheap. You want both, because a correct retry that costs a full run every time is a reliability fix you will be tempted to turn off the first time the token bill arrives.

Some steps cannot be made idempotent. Compensate them.

Not every side effect has a key you control. An idempotency key works when the tool, or the service behind it, will honor it and dedup on your behalf. Plenty of real actions will not. A third-party API that charges a customer's card, a government portal that accepts a filing, a partner system that dispatches a shipment: some of these expose no idempotency mechanism, and some take an action in the physical world that a later call cannot make un-happen. For those, replay-safety is off the table, and you need the other tool.

That tool is the compensating action, the core of what distributed systems call a saga. You model the run as a sequence of steps where each step that takes an irreversible action is paired with a second action that cancels its effect: the charge is paired with a refund, the filing with a retraction, the dispatch with a recall. You record, durably, which steps actually completed. When the run fails partway, you do not roll back, because there is nothing to roll back to. You run the compensators, in reverse order, for exactly the steps that completed. This is the crucial mental shift. A rollback is the system pretending the action never happened. A compensation is a new, forward action that admits the first one happened and undoes its effect. The charge was real. The refund is also real. Both show up in the ledger, and that is correct, because both are things that actually occurred. Reserve compensation for the steps that genuinely cannot be made idempotent, because it is more machinery than a key, and prefer idempotency wherever the tool will let you have it.

Not every failure should be retried

Retrying assumes the failure was bad luck. Often it was not. A rate limit, a socket timeout, a transient 503 from a model provider: these are worth retrying, because the same call a moment later has a good chance of succeeding. A malformed structured output that fails schema validation, a tool that returns "recipient address is invalid," an argument the model got wrong: these are deterministic. The exact same call will produce the exact same failure every time, so a retry is not a second chance, it is the identical mistake at a later timestamp, burning tokens and latency to arrive at the same error. This is the brakes chapter's loop guard wearing a different hat: retrying a deterministic failure is a two-iteration version of the loop that never makes progress.

So classify before you retry, at the tool boundary where the error is raised. Transient failures get a bounded retry with backoff. Deterministic failures fail fast, stop the run, and escalate to a human or a repair step, because no amount of retrying will change the answer. A system that retries everything the same number of times treats a network blip and a schema violation identically, and pays full retry cost on the half of failures that were never going to resolve. Knowing which kind of failure you are looking at is the difference between a retry that recovers and a retry that just delays the inevitable.

Do the math on the failure surface

The reason this matters more in a multi-agent system than a single call is not that the individual calls are less reliable. It is that there are so many more of them, and a run needs all of them to land. Take the run concretely: six agents, each making one model call, plus roughly eight tool calls spread across them, so about fourteen independent calls that all have to succeed for the run to complete cleanly. Give each call a modest one percent chance of a transient failure. The probability that all fourteen succeed is 0.99 to the fourteenth power, which is about 0.87. So roughly thirteen percent of runs, better than one in eight, hit at least one failure somewhere in the chain.

Compare the single call: one call at one percent is a one percent failure rate, and a safe retry clears it. The multi-agent run is about thirteen times more likely to land in a partial state per run, purely from having fourteen calls instead of one, and every one of those partial states is a live question, does a retry duplicate a side effect or resume cleanly. At a thousand runs a day, thirteen percent is about a hundred and thirty runs landing halfway every day. Without idempotency and checkpointing, each of those is a potential double-sent email or a full and expensive re-run. With them, each is a cheap resume that the outside world never notices. Notice what the fix changed and what it did not. The failure rate is still thirteen percent, because you did not make the calls more reliable. What you changed is the cost of a failure, from a duplicated irreversible action down to a few thousand tokens of replay. In a system that fails one run in eight, that is the number that decides whether you can run it in production.

A single call fails all at once, so a retry is safe. A multi-agent run fails in the middle, after it has already touched the outside world, so a retry is the second bug. Reliability is not adding retries. It is making the thing a retry replays safe to replay.

When you do not need any of this

The honest caveat, as in every chapter of this series. If no agent in your run takes an irreversible side effect, if the whole pipeline only reads, reasons, and returns text to a person who decides what to do with it, then a failure just means run it again, and a blind retry is perfectly safe because there is nothing to duplicate. A pure analysis pipeline that never sends, writes, files, charges, or submits does not need idempotency keys or sagas, and the single-agent baseline is again the cheapest thing that could possibly work. The entire apparatus earns its cost at one specific moment: when an agent takes an action that touches the outside world and cannot be un-taken. For the document-heavy systems this series is about, systems that file, notify, and write to a client's system of record, that moment is the whole point of building them. You cannot tell a filing pipeline to stop filing. You can only make sure that when it fails halfway, and one run in eight will, it does not file the same thing twice.

We build production multi-agent systems for document-heavy work in legal, finance, and insurance, where agents file, notify, and write to systems of record, so a run that fails halfway is a real filing with real liability, not a lost request you can quietly retry. If your agents take any action that touches the outside world and cannot be un-taken, the partial-failure path is already there, whether or not it has bitten you yet. Talk to an engineer or see how we work.