TL;DR
Classic RAG retrieves once and answers. Agentic RAG lets the model decide whether to retrieve, what to ask for, and when to stop. Here is what that costs.
Classic RAG does one thing in one order: embed the question, fetch the nearest chunks, put them in the prompt, answer. It works, and it fails in a way that is easy to recognise once you have seen it. The retrieval was wrong, the model did not know that, and it answered confidently anyway.
Agentic RAG changes one thing. Retrieval stops being a step in a pipeline and becomes a tool the model can choose to call. Everything else follows from that.
For engineers deciding whether to add the loop. 8 minute read.
What actually changes between RAG and agentic RAG?
| Classic RAG | Agentic RAG | |
|---|---|---|
| When retrieval happens | Always, once, before generation | When the model decides it needs it |
| What gets retrieved | The user's question, embedded as-is | A query the model wrote for the purpose |
| Number of retrievals | One | As many as the loop allows |
| If retrieval returns nothing useful | The model answers anyway | The model can notice and try again |
| Failure mode | Confident answer from wrong context | Loop that does not terminate |
| Latency | One retrieval, one generation | Unbounded until you bound it |
The row that matters is the fourth. In classic RAG, nothing in the system is responsible for asking whether the retrieved context was any good. The chunks arrive, they go in the prompt, and the model treats them as the truth of the matter.
It is worth separating this from GraphRAG straight away, because the two get used interchangeably and they are not alternatives. GraphRAG changes what you retrieve over by giving the corpus a structure. Agentic RAG changes how many times and on what terms you retrieve. You can run either without the other, and a system with a hard retrieval problem often needs the first rather than the second.
Why does single-shot retrieval fail?
Three ways, and they are not exotic.
The question is not the query. A user asks "why did our margin drop in Q3". The useful documents do not contain that sentence. They contain discount approvals, a price list revision, and a supplier invoice. Embedding the question and fetching the nearest neighbours retrieves things that sound like the question rather than things that answer it.
The answer needs two hops. "Which of our suppliers are affected by the new packaging rule" requires finding the rule, then finding the suppliers, then intersecting. One retrieval cannot do this. It will return the rule, and the model will guess at the rest.
The corpus does not contain it. The honest answer is "this is not in your documents". Single-shot RAG has no mechanism to reach that conclusion, because retrieval always returns something, and the nearest chunks are still the nearest chunks when the answer is absent.
What decisions does the agent actually make?
This is the part worth being precise about, because "agentic" is used loosely enough to mean nothing.
An agentic retrieval loop adds up to four decision points:
- Route. Does this question need retrieval at all? "Summarise the text I just pasted" does not. Neither does "what is 12% of 40,000". Routing away from retrieval is a cost saving and an accuracy saving at once.
- Formulate. What should the query actually be? Rewriting the user's question into one or more retrieval queries is where multi-hop questions become answerable.
- Assess. Are these chunks relevant? A grading step between retrieval and generation is the mechanism that classic RAG is missing.
- Iterate or stop. If the context is thin, retrieve again with a different query. If it is still thin, say so.
You do not need all four. Most production systems that call themselves agentic implement two, usually formulate and iterate.
What does the loop cost?
Everything you add here you pay for three times: in latency, in tokens, and in the number of ways the system can fail.
| Cost | Classic RAG | Agentic RAG |
|---|---|---|
| Model calls per answer | 1 | 1 per decision point, plus 1 per retry, plus the answer |
| Tail latency | Bounded by one retrieval | Bounded only by your iteration cap |
| New failure modes | Bad context | Loop that will not terminate, query drift, retry on a query that was fine |
| What you must now monitor | Answer quality | Answer quality, iteration count, cost per answer |
Query drift is the one that surprises people. Each reformulation moves the query further from what the user asked. By the third rewrite the system can be retrieving competently against a question nobody posed.
The mitigations are unglamorous and they are the whole job: cap iterations, keep the original question in context at every step, and log the query at each hop so drift is visible rather than inferred.
When is agentic RAG the wrong build?
Four cases where the honest answer is to keep the pipeline simple.
Your retrieval is not the problem. If answers are wrong because the chunks are badly split or the documents are stale, an agent will simply retrieve the same bad chunks more times. Fix chunking and freshness first. This is the most common misdiagnosis in the category.
The questions are homogeneous. A support assistant answering variations of forty known questions does not need a model to decide whether to retrieve. It needs good retrieval and a cache.
Latency is the product. If a human is waiting on the answer inside a conversation, an unbounded loop is a worse experience than an occasionally wrong single-shot answer, and users forgive the second more readily than the first.
Nobody is measuring. An agentic loop without evaluation is strictly worse than a pipeline without evaluation, because it has more places to go wrong and the same amount of visibility. Which is none.
How do you know it is working?
Not by reading answers. By measuring the thing the loop was added to fix.
The single most useful number is the share of questions where retrieval was assessed as insufficient and the system said so, rather than answering anyway. Classic RAG scores zero on this by construction. If your agentic system also scores near zero, the assessment step is decorative.
After that: iteration count distribution (a long tail means drift or a missing stop condition), cost per answered question, and answer quality held against a fixed question set that does not change when the system does.
The point of the loop is not more retrieval. It is a system that can tell the difference between having the answer and not having it.
Share this article
Enterprise GraphRAG & Knowledge Systems
Topic Cluster
How we design the graph schema, the retrieval path and the evaluation harness behind these articles.
