Topical authority hub

Multi-Agent Systems in Production

When many agents beat one and when they do not: the decision gate, orchestration topologies, coordination, cost, and how to evaluate the result.

multi-agent systems

Articles: 11
Total read: 152m
Pillar: Set

Start here — foundational guide

PillarFoundational guide

Most Multi-Agent Systems Would Work Better as One Agent

A team splits its working agent into a planner, three researchers, a critic, and a synthesizer. Latency triples, the bill jumps tenfold, and the researchers return three contradictory answers because none saw what the others found. Multi-agent architecture is a cost you pay for two things, parallelism on independent subtasks and isolation for specialists, and most teams pay it without getting either. It multiplies tokens, multiplies coordination failures, and compounds unreliability across every hop. Default to one capable agent; reach for many only on a clear gate, and then share structured state, route instead of fanning out, and keep writes single-threaded.

Mudassir Marwat17 minJul 1, 2026

Read the guide

Continue the path

Ordered by chapter. Each post stands alone but builds on the one before it.

Chapter 1

Four Ways to Wire a Multi-Agent System (and When Each One Breaks)

You settled the question of whether to use multiple agents. Now comes the choice that matters more than the head count: how to wire them. There are four topologies, a sequential pipeline, an orchestrator with parallel workers, a hierarchy of supervisors, and a peer-to-peer network, and each fits one shape of work and hides one failure. The same three agents at eighty percent each are about fifty-one percent reliable wired to all-must-succeed, about ninety percent as a majority vote, and about ninety-nine percent when any one can and you can verify it. Topology, not head count, sets your reliability. Underneath it, share structured state instead of messages, keep the writes single-threaded, and read the shape off the task dependency graph.

17 minRead

Chapter 2

When a Multi-Agent System Fails, Which Agent Broke?

You decided to use multiple agents, and you wired them. Now the system gives a confident wrong answer and hands you no stack trace, and the hardest production question arrives: does this work, and when it does not, which agent broke? Single-agent evaluation does not transfer, because a multi-agent system fails in the seams. You need three layers: outcome tells you whether it failed, component tells you which parts work and gives you the per-agent reliability, and trajectory tells you where in the flow it broke. Put them together and the reliability math becomes a diagnostic: if the parts predict about seventy-three percent and the system delivers fifty-five, the gap is an interaction bug. And because these systems are non-deterministic, one green run is not a passing grade, so you measure a rate over many runs off a trace you can actually read.

18 minRead

Chapter 3

Your Agents Don't Share a Brain. They Pass Notes.

You wired the agents, and you learned the failures live in the seams between them. This chapter names the seam: it is the hand-off. When one agent finishes and the next begins, everything the first learned, its documents, its discarded hypotheses, its doubt, has to survive a trip through a single message, and the next agent acts on that message alone. Your agents do not share a brain, they share a mailbox, and every message is a lossy compression of what the sender knew. Four things go wrong: context loss, semantic drift, error laundering, and a coordination tax that grows with the square of the chain length. The fix is not a better prompt. It is a shared, typed store the agents read and write, plus hand-off contracts that make a bad message fail loudly instead of quietly.

10 minRead

Chapter 4

Your Multi-Agent System Has No Brakes

You wired the agents and gave them a shared place to coordinate. Now, what stops them? In most multi-agent systems no single component owns the decision of what runs next and when to stop, so control is emergent, and emergent control does not reliably halt. It fails four ways: non-termination, premature termination, mis-routing, and a runaway cost tail where the median run is fine but the one-in-fifty tail run costs twenty to fifty times more. This chapter builds the brakes: a termination predicate the system checks instead of the agent's self-report, a hard global budget that bounds worst-case cost, one controller that owns routing, and loop detection that reads the shared store. Architecture is the engine. Control is the brakes.

9 minRead

Chapter 5

Your Multi-Agent System Is a Black Box

You gave the agents brakes in the last chapter. Brakes stop a runaway, but they do not tell you which agent is dragging, what a run costs, or why last night's batch tripled. Most multi-agent systems ship to production as a black box: the team can see that a run happened and returned, and nothing in between. Worse, agent failures are silent, they return a 200 OK with a confident wrong answer, so uptime and error rate read green while the output is wrong. This chapter builds the instrument panel: the trace tree as the unit of observability, per-span cost attribution that finds the one agent owning two-thirds of the bill, structural sampling that keeps every run worth investigating, and continuous semantic evaluation, the only signal that catches a wrong answer no system metric will flag. Brakes stop a runaway. Instruments show it coming.

11 minRead

Chapter 6

One Poisoned Agent Poisons the Chain

You gave the agents brakes, then instruments. Neither stops a poisoned document from turning the pipeline against you. A single model call has one trust boundary, trusted system prompt versus untrusted input; a multi-agent system erases it, because every agent's output becomes the next agent's trusted input. So an injection planted in one document read by one agent propagates through the hand-offs to the agent that can send email or write to a system of record. The lethal trifecta, untrusted content, private data, and the ability to communicate out, gets split across your agents and recombined by the chain, so a per-agent review clears every agent and still misses the exploit. You cannot filter your way out. What works is structural: least privilege so the reader holds no dangerous tool, quarantine untrusted content behind a tool-less model, and a deterministic guardrail on the dangerous action. Tracing shows the attack. It does not stop it.

12 minRead

Chapter 7

Your Retry Just Sent the Email Twice

You secured the tool boundary against an attacker. Now the tool fails on its own, with no attacker in sight: a model call times out halfway through the run, a retry kicks in, and the send-email tool that already fired once fires again. A single model call fails atomically, so retrying the whole call is safe. A multi-agent run is fourteen calls with no transaction around them, so it fails in the middle, after some agents have already sent the email or written the row, and there is no rollback. A blind retry replays the steps that succeeded and duplicates their side effects, so the retry is not recovery, it is a second bug. What works is structural, at the action boundary: make every side-effecting tool idempotent with a key so a replay is a no-op, checkpoint each step so a retry resumes instead of restarts, and compensate the steps you cannot make idempotent. Across fourteen calls, one run in eight lands partial. The failure rate does not change. The cost of a failure does.

13 minRead

Chapter 8

You Pay for the Same Context Fourteen Times

You made a run reliable and secure. Now look at what it costs. A single model call has an obvious, fixed cost, paid once. A multi-agent run is dominated by input tokens, and in a naive pipeline every agent re-ingests the accumulated context, so you pay to re-read the same system prompt, tools, and documents on every one of the roughly fourteen calls. The bill scales with context-per-call times number-of-calls, not with agent count, so six agents cost closer to fifteen times a single call, not six. Cost control is structural: minimize the context at every handoff, tier the models so the mechanical agents run cheap, cache the static prefix that repeats every call, and do not fan out when a single call will do. Stacked, they take a naive run from about forty-one thousand frontier-priced tokens to roughly a seventh of the cost for the same output. The most effective lever is minimization, because a token you never send is free at every tier, in every cache, and on every retry.

13 minRead

Chapter 9

There Is No Such Thing as a Local Change

You made a run cheap. Now try to change it. In a multi-agent system there is no such thing as a local change, because every agent output is the next agent input and the contract between them is natural language, not a typed schema, so nothing stops a change from spreading. Edit one agent prompt, swap its model, or change a tool, and you have moved the behaviour of every agent downstream, which was tuned on the old output. The unit of change is the whole pipeline, not the agent, so you cannot ship it component-by-component the way service intuition says. Version the pipeline as one artifact, keep golden transcripts and replay every change against them, canary on live traffic before you promote, and pin the model version so the vendor cannot change your pipeline under you. The sneakiest change of all is the cost move of tiering an agent down to a cheaper model, because it rewrites the input of the next agent while looking like a harmless config tweak.

15 minRead

Chapter 10

You Built a Line When the Work Was a Graph

You made the run cheap and safe. It is still slow, and not because any one agent is slow. Your latency is not the sum of your agents. It is the longest path through the graph of what depends on what, the critical path, because two agents that do not read each other output can run at the same time. Teams build the pipeline as a straight line, so a six-agent chain at about two seconds an agent takes twelve seconds when the real dependency graph allows eight. The fixes are structural. Draw the dependency graph not the flowchart, parallelize the independent branches so the two extractors run at once, collapse the hops that do not earn their round-trip, stream the last agent so time-to-first-token is short, and cap the tail, because a six-hop chain slow case compounds with every hop. One catch: latency and cost pull in opposite directions, so speed bought by running more agents at once is often paid for in tokens.

17 minRead

Build it for real

Read the writeup. Now ship the system.

Cognilium engineers ship the architectures behind these articles for enterprise teams. If you're mid-build on multi-agent systems in production, talk to us.