You Built a Line When the Work Was a Graph

Your six agents run one at a time, so you pay the sum of their latencies. But the work underneath is a graph, and your real latency is the longest path through it. Find the branches that can run at once, and stop waiting on hops that were never on the critical path.

The last two chapters were about the two numbers your finance team and your on-call engineer feel: the bill and the blast radius. Cost was the bill, and we cut it by paying for the same context once instead of fourteen times. Change management was the blast radius, and we made it safe to touch a pipeline where every agent's output is the next agent's input. This chapter is the number your user feels, the only one they actually experience while they sit and watch the tab spin: the clock.

Here is the uncomfortable part. The pipeline you just made cheap and safe is still slow, and it is slow for a reason that has nothing to do with any single agent being slow. Every agent in it might be fast. The system is slow because it runs one agent at a time, and you built it that way without deciding to, because the flowchart on the whiteboard was a straight line and you wired the code to match the picture. The picture lied. The work was never a line. It was a graph, and you have been paying the length of the line when you only owed the length of the graph's longest path.

TL;DR

In a multi-agent system your latency is not the sum of your agents' latencies. It is the longest path through the graph of what actually depends on what, the critical path, plus the overhead of every hop and the compounding risk in the tail. Teams pay far more than that because they build the pipeline as a straight sequence, where each agent waits for the previous one to finish even when it did not need to, so a six-agent chain at roughly two seconds an agent takes about twelve seconds when the real dependency graph might allow eight. The fixes are structural, not per-agent. Draw the dependency graph and not the flowchart, because two agents that do not read each other's output can run at the same time. Parallelize the independent branches, so the two extraction workers run at once and you pay the slower of the two, not the sum. Collapse the hops that do not earn their round-trip, because every agent boundary is a full model round-trip and two agents that always run in sequence and could be one prompt are two round-trips where one would do. Stream the final agent's output, because perceived latency is not actual latency and time-to-first-token is what the user grades you on. Speculate and cache on the critical path, starting the likely-next agent before the current one commits and caching the steps that are deterministic. And optimize the tail rather than the average, because a six-hop chain's p99 is not any one agent's p99, it is the chance that any hop on the critical path hits its slow case, and that chance compounds with every hop you add. One more thing, because it is the part nobody warns you about: latency and cost pull in opposite directions, so the speed you buy by running more agents at once is often paid for in tokens.

Your six agents run one at a time

Start where the time actually goes. Take the six-agent extraction pipeline this series has used throughout: a planner that routes the work, a retriever that fetches the source documents, two extraction workers that pull fields against a schema, a reconciler that merges their outputs and resolves conflicts, and a reviewer that makes the final judgment and holds the two tools that touch the outside world, finalize to the system of record and send email. Suppose each of those agents takes about two seconds for its model call, which is a reasonable figure for a mid-size model generating a few hundred tokens. Wire them as a straight line, each waiting for the one before it, and the run takes about twelve seconds. Your user waits twelve seconds.

Now notice what those twelve seconds are actually made of, because it is not compute in the sense you are used to. Almost none of it is your code running. Each hop is a full round-trip to a model: the network out, the queue at the provider, the time to first token, the generation of every token in sequence, the network back. An agent is not a function that returns in microseconds. It is a remote call that takes seconds, and a six-agent pipeline is six of those remote calls stacked end to end. This is why multi-agent latency feels different from ordinary service latency and why the usual instincts fail. In a normal service you profile to find the one slow function and you speed it up. Here there is no one slow function. There are six unavoidable round-trips, and the only lever that matters is how many of them you are forced to wait on in a row.

Latency is the longest path, not the total work

Here is the idea the whole chapter turns on. Your latency is not the total work the pipeline does. It is the longest path through the graph of what depends on what. Add up every agent's time and you get the total work, twelve seconds in our example. That number tells you the cost, roughly, but it does not tell you the latency, because latency only counts the work you cannot avoid doing in sequence. If two agents do not depend on each other, they do not have to wait for each other, and the time you pay is the slower of the two, not their sum. So the real question is never "how much total work does a run do." It is "what is the longest chain of steps where each one genuinely needs the output of the one before it." That chain is the critical path, and the critical path is your latency floor. Everything off the critical path is free time you are currently paying for by accident.

Look at the six agents again with that lens. The planner runs first, and everything depends on how it routed, so it is on the critical path. The retriever runs next, and the extractors need the documents it fetched, so it is on the critical path too. But then come the two extraction workers, and here is the whole point: the second extractor does not read the first extractor's output. They both read the retriever's documents and they both write to the reconciler. They are independent. In a straight line they run one after the other and cost you four seconds. In the real dependency graph they are a fork, they run at the same time, and they cost you the slower of the two, about two seconds. You just deleted two seconds from the run without touching a single prompt, a single model, or a single line of any agent's logic. You deleted it by noticing that the line you drew was not the graph the work actually forms.

Draw the dependency graph, not the flowchart

The flowchart on your whiteboard shows sequence: this agent, then that agent, then the next. It is a picture of the order you happened to write the code in, and it quietly becomes the order the code runs in. What you need instead is the dependency graph, which is a different picture that answers a different question: for each agent, whose output does it actually read. Draw that, and the parallelism falls out for free, because any two agents with no path between them can run at the same time. This is not a refactor you do once. It is a way of reading your own architecture, and the shape of that architecture, whether it is a chain, a tree, or a wide fan-out, is precisely what sets your critical path.

The discipline is to interrogate every edge in the line and ask whether it is a real data dependency or an accident of authoring. The reviewer needs the reconciler's output, so that edge is real. Does the retriever need the planner's full output, or just the query the planner produced, which might be ready before the planner has finished explaining its reasoning? Does the reconciler need both extractors completely finished, or can it start merging the fields that are already in as they stream? Most pipelines are drawn as a single line because a line is easy to draw and easy to reason about, but the underlying data dependencies almost always form a wider graph with slack in it. Your job is to find the slack. You will not find it by staring at the code, because the code encodes the sequence you are trying to escape. You find it by drawing the dependencies as their own graph and looking for the agents that share no ancestor.

Parallelize the branches that do not depend on each other

Once you can see the independent branches, running them at once is the highest-leverage change you can make, and it costs you nothing in quality because the agents were never going to influence each other anyway. Fan out the two extraction workers so they run concurrently against the same documents, and fan in at the reconciler, which waits for both and then merges. The run pays the slower extractor instead of the sum, and the wider your fan-out, the bigger the win: three independent extractors run as a straight line cost you six seconds and run in parallel cost you two.

There are two details that decide whether this actually helps in production. The first is the join. A fan-in waits for the slowest branch, so parallel work is only as fast as its slowest member, which is the reason the tail matters so much and why we come back to it below. The second is that parallelism has a floor set by your dependencies and no amount of engineering lowers it. If every agent genuinely reads the previous agent's output, the graph is a true line, the critical path is the whole thing, and there is nothing to parallelize. That is a signal in itself. A pipeline that is a genuine unbroken chain of dependencies is often a pipeline with more agents than the task needs, because real work usually has some independent parts, and if yours has none, it is worth asking whether some of those sequential hops are really separate agents or one agent you split for no reason.

Some hops do not earn their round-trip

Parallelizing removes waiting between agents that run at the same time. Collapsing removes agents. Every hop in the pipeline is a full model round-trip with its own latency floor, its own queue, its own network cost, and its own place in the tail, so the cheapest second is the one you delete by not making the call at all. Look for two agents that always run one immediately after the other, where the first does nothing but hand its output straight to the second, and ask what the boundary between them is buying you. Sometimes the answer is real: they run on different models, or the first agent's output is reused by something else, or you keep them separate so a failure in one is isolated from the other. Keep those. But often the boundary is buying you nothing except a second round-trip, and two agents that could have been one prompt are latency you are paying for a distinction that does not exist in the work.

Be honest about the direction this cuts, because it is the opposite of the advice you have heard everywhere else. The whole series has pushed you toward fewer, clearer agents, and this is the same push from the latency side. A planner whose only job is to emit a query, immediately followed by a retriever, might be one step where the model retrieves as a tool call inside a single turn. A reconciler and a reviewer that always run back to back, where the reconciler merges and the reviewer rubber-stamps, might be one agent that merges and judges in one pass. Collapsing is not always right, and the handoff between two agents sometimes carries real value that a merged prompt would lose. But every hop should have to justify its round-trip, and the ones that cannot are the fastest latency you will ever recover, because you recover the whole hop, tail and cost included, not just the waiting around it.

Stream, so the user stops waiting on the whole run

Everything so far shortens the actual run. Streaming shortens the run the user feels, which is a different number and often the one that matters more. A user does not experience your pipeline as a duration. They experience it as a wait for the first useful thing to appear on the screen, and if nothing appears for twelve seconds and then everything appears at once, that is a worse experience than if the answer begins forming at eight seconds and finishes at thirteen, even though the second run is technically slower. Perceived latency is set by time-to-first-token, not time-to-last-token, and the final agent in most pipelines can stream.

The move is to stream the reviewer's output token by token as it generates, so the user sees the answer taking shape instead of watching a spinner. You can go further and surface the intermediate agents' progress, showing that the documents were retrieved, that extraction is running, that reconciliation is underway, so the wait is legible instead of blank. None of this makes the pipeline finish sooner, and that is the point: it decouples what the user endures from what the clock measures. It is also the cheapest optimization in this chapter, because it changes no dependencies and no models, only how you deliver the result you were already going to produce.

Your users do not grade you on how long the pipeline runs. They grade you on how long they stare at nothing. Those are different numbers, and streaming is how you attack the one that hurts.

Speculate and cache on the critical path

Once the critical path is as short as your dependencies allow, the remaining latency lives inside the hops on it, and there are two ways to attack a hop you cannot delete. The first is to start it early. If the planner almost always routes document-heavy work to the same retrieval, you can begin fetching the likely documents speculatively while the planner is still finishing, and throw the work away on the rare occasions the planner routes elsewhere. Speculative execution trades wasted work for wall-clock time, so it pays off exactly when the guess is usually right and the wasted call is cheap. The second is to not run the hop at all. Any agent whose output is a deterministic function of its input can be cached, so the second identical run returns instantly, and retrieval in particular is often cacheable because the same documents get pulled again and again. This is the same reuse idea from the cost chapter seen from the latency side: a cache hit saves both the dollars and the seconds of a round-trip you have already paid for once.

Both of these need something you may not have yet, which is per-hop timing. You cannot shorten the critical path if you do not know which hops are on it and how long each one takes, and that number comes from the same tracing your observability already produces, read as a latency breakdown per span instead of a cost or error breakdown. Attribution here is the same problem as attribution when a run fails: when the pipeline is slow, which agent was slow, and is it on the critical path or off it. Optimizing a hop that runs in parallel off the critical path speeds up nothing, and without per-span timing you cannot tell the difference, so you end up tuning agents that were never the bottleneck while the real one sits untouched.

Optimize the tail, not the average

Here is the number that will hurt you in production, and it is the one almost nobody computes in advance. Your pipeline's latency is not its average latency. It is its tail, the p95 and p99, the slow runs that a real user will hit often enough to remember, and the tail of a multi-agent chain is far worse than the tail of any single agent in it. The reason is compounding. Suppose each of your six agents is fast almost always but has a five percent chance of hitting a slow path, a long generation, a provider hiccup, a retry. On a single agent that is a rare event. Across a six-hop chain where every hop is on the critical path, the chance that the whole run is clean is 0.95 to the sixth power, which is about 0.735, so more than one run in four hits at least one slow hop. Add hops and it gets worse fast: at ten hops it is about one run in three. The tail does not add across a chain. It compounds, and every agent you put on the critical path multiplies the probability that some user's run is the slow one.

This changes what you optimize. Bringing the average hop down from two seconds to 1.8 barely moves the experience, because the runs that hurt were never the average runs. What moves the experience is capping the tail: a timeout on each hop on the critical path, so a single slow model call cannot hold the whole run hostage, and a fallback for when the timeout fires, a smaller faster model, a cached answer, a degraded but complete result. This is the same discipline as the reliability chapter, where a timeout with no fallback is just a different way to fail, applied to speed instead of correctness. The goal is not to make the fast runs faster. It is to make the slow runs less slow and less frequent, because the tail is what your users actually feel and the average is a number that comforts only you.

Latency and cost pull in opposite directions

Now the part you did not ask for, because it is the part that turns a clean win into a real decision. Speed and cost are not free of each other. Several of the moves in this chapter buy you latency by spending money. Parallelizing the two extractors makes the run faster, but if you were relying on the first extractor sometimes making the second unnecessary, running them both always costs you tokens you used to save by stopping early. Speculative execution buys wall-clock time with wasted calls, and every guess that turns out wrong is money spent on a result you threw away. Streaming is close to free, but the retries and fallbacks that cap the tail mean running a second model call on the runs that time out, which is real spend on your slowest traffic. The cheap-and-fast dream is a small model on every hop, but the cost chapter already showed that the small model changes behaviour, so speed bought by tiering down is a change that has to be earned like any other.

The one move that saves both is collapsing hops, because deleting a round-trip removes its seconds and its tokens at the same time, which is why it sits at the top of the list even though it is the hardest to do without losing something. Everything else is a trade you have to price. The way to price it is to decide, per pipeline, which number you are actually optimizing. A user-facing pipeline where someone waits on the result should spend money to buy latency, within reason, because the wait is the product. A batch pipeline running overnight on a queue should not spend a cent to be faster, because nobody is waiting and the only number that matters is the bill. Most teams get this backwards, optimizing the cost of the interactive path and the latency of the batch path, because they never separated the two and asked which clock each one runs on.

When you do not need any of this

None of this applies to a single model call, which has no downstream and therefore no critical path to shorten, and it barely applies to a genuinely short pipeline of two or three agents that already returns fast enough for its use. The entire chapter is a tax that scales with the depth of your graph and the number of people waiting on it, and if your graph is shallow or nobody is waiting, do not pay the tax. A pipeline that runs overnight on a batch queue can be as slow as it likes, because the only clock it answers to is the cost clock. A pipeline behind a human reviewer who reads each result for a minute before acting does not benefit from shaving three seconds off a twelve-second run, because the human is the slow step and always will be. Optimizing latency there is effort spent making a number smaller that no user will ever feel.

Reach for this chapter when a person is waiting on a synchronous result and the pipeline is deep enough that the wait is real. That is when the critical path is worth drawing, when the independent branches are worth parallelizing, and when the tail is worth capping. And notice that the first and cheapest latency optimization is the one from the very start of this series: the fewest agents the task genuinely needs. Every agent you add is another hop on some path, another round-trip, another five percent of tail risk, another edge that might or might not be on the critical path. The shortest pipeline is the fastest pipeline, and it is also the cheapest and the safest to change, which is the whole argument of this series arriving at the same place from a third direction.

We build production multi-agent systems for document-heavy work in legal, finance, and insurance, where a pipeline runs thousands of extractions a day and a user is waiting on each result, so the difference between a twelve-second run and an eight-second one is the difference between a tool people use and one they abandon. The thing that keeps those systems fast is not a faster model on every hop. It is structure: the shortest critical path the work allows, the independent branches run in parallel, the hops that earned their round-trip and no others, and a capped tail so the slow runs stay rare. If that is the kind of system you are trying to run, talk to an engineer or see how we work.