You Cannot Retrieve Your Way Out of a Bad Graph

In a GraphRAG system, answer quality is decided at construction, not retrieval. If extraction left you duplicate entities, untyped edges, and no provenance, no amount of retrieval tuning will save you. The graph is built once and queried forever, so build it like it matters.

A team comes to us with a GraphRAG system that gives wrong answers, and they have spent three weeks on retrieval. They have tuned the number of chunks they pull, added a reranker, weighted the hybrid search, tightened the prompt that assembles the context. The answers are still wrong, and they are wrong in a specific way: the system confidently connects things that are not connected and misses connections that are. So they conclude retrieval is still not good enough and reach for a better reranker. It will not help. The retrieval is fine. The graph it is retrieving over is the problem, because the graph was built by a single loose extraction prompt that produced three copies of every company and edges nobody typed. You cannot retrieve your way out of a bad graph.

This is the mistake that defines the difference between vector RAG and GraphRAG, and almost nobody sees it coming, because it does not exist in the vector world they came from. In vector RAG the index is forgiving. Chunks are independent, a bad chunk is one bad result among many, and if the index is wrong you re-embed it in an afternoon. A knowledge graph is not forgiving. It is a stateful, accumulating artifact, and a wrong edge is not one bad result, it is a false fact that every query traversing that edge will inherit, forever, until someone detects and repairs it. In GraphRAG, the quality of your answers is set at construction time, and retrieval only reads back what construction wrote.

TL;DR

In a GraphRAG system, answer quality is decided at construction, not retrieval, so tuning the retriever while the graph is broken is effort spent in the wrong place. A vector index is forgiving because chunks are independent and cheap to rebuild; a knowledge graph is unforgiving because it is a stateful artifact where one wrong edge, one duplicate entity, or one missing provenance link corrupts every query that traverses it, and the error compounds through multi-hop traversal. Six disciplines make construction reliable. Design the schema before you extract, because typed nodes and typed edges are an engineering decision, not something you let a language model invent per document. Treat extraction as a pipeline, not a prompt: a real production pipeline runs parse, classify, evidence, extract, validate, score, orchestrate, and graph-write as eight distinct stages, each with its own job and failure mode. Resolve entities at write time, so one company with eleven surface names becomes one node with a canonical ID before it lands in the graph, not a read-time patch you apply forever after. Make provenance first-class, so every node and edge carries the source document, the source span, and a per-field confidence, which is what makes grounding, audit, and user trust possible at all. Validate at insertion or pay at retrieval, because a write-time gate that rejects orphans, duplicates, untyped edges, and low-confidence claims is the difference between a graph and a slow-motion garbage fire. And test construction with a real multi-hop query: if the graph was built right, a three-hop question answers itself, and if it was built wrong, no retriever will recover the answer.

The graph is built at extraction, not retrieval

Start with where the quality actually lives, because it is not where people look. A GraphRAG system has two halves that feel symmetric but are not. There is construction, the offline process that reads your documents and writes a graph, and there is retrieval, the online process that reads the graph and assembles a context for the language model. Teams pour their attention into retrieval because retrieval is visible: it runs on every query, it has knobs you can turn, and when an answer is wrong the retriever is the last thing that touched it. Construction is invisible. It ran once, last week, in a batch job nobody watches, and its output is a database you rarely look at directly. So the graph gets built fast and loose, and then retrieval spends the rest of its life trying to compensate for a foundation it cannot change.

The asymmetry is the whole point. In vector RAG the two halves really are loosely coupled, because the index is a bag of independent chunks. If a chunk is badly split or poorly embedded, it costs you that one chunk, and re-chunking or re-embedding the corpus is a cheap, idempotent operation you can run whenever you like. A graph is coupled to itself. Every node participates in edges, every edge is a claim that some query will traverse, and traversal is multiplicative: a three-hop query walks three edges, and if any one of them is wrong the answer is wrong, no matter how good the other two are. That is why a graph that is ninety-five percent correct at the edge level can still be wrong on most multi-hop queries, because the errors do not average out, they compound along the path. The retriever inherits every one of those errors and has no way to know they are there, because a false edge looks exactly like a true one. This is also the mechanism behind graph rot, the slow decay that makes a knowledge graph lie to your AI: rot is just construction errors accumulating faster than anyone repairs them.

Why teams tune retrieval and ignore construction

It is worth naming why the wrong half gets all the attention, because the reasons are structural and you will fight them on every project. Retrieval is legible and construction is not. When a user reports a bad answer, you can replay the query, watch the retriever pull its chunks, and see something concrete to adjust. You cannot replay construction against a single bad answer, because the fault was written into the graph in a batch job that processed ten thousand documents at once, and the specific edge that ruined this answer is one row among millions. So the feedback loop points at retrieval. Every bad answer feels like a retrieval bug, because retrieval is the part you can see failing.

Retrieval also gives faster, more satisfying wins, which is exactly why it is the trap. Turning a knob and watching an answer improve on your test query feels like progress, and sometimes it is, but a retrieval win on the queries you happen to test can mask a construction problem that is quietly wrong on the queries you do not. The reranker that fixes your ten demo questions does nothing for the entity that got split into three nodes, because no reranker can merge nodes the graph believes are different companies. Meanwhile construction work is slow, upstream, and invisible, and it improves answers you were not looking at, which makes it feel less urgent even though it is where the leverage is. The discipline is to treat a wrong multi-hop answer as a construction hypothesis first and a retrieval hypothesis second, which is the opposite of where instinct points.

Design the schema before you extract

The first construction decision is the schema, and it is an engineering decision that belongs to your team, not a thing you let a language model improvise document by document. A property graph is typed: nodes have types, edges have types, and those types are the vocabulary every query is written against. If you let extraction invent types as it goes, you get a graph where the same relationship is called `owns`, `controls`, and `has_stake_in` on three different documents, and no query can find all of them, because there is no single edge type to traverse. The schema is the contract between construction and retrieval, and like any contract it has to be decided up front and enforced, not discovered after the fact.

Make it concrete with a real one. On a family-office investment platform we built, the graph has six node types, Company, Investment, Document, Extraction, Person, and Vehicle, and five edge types, EXTRACTED_FROM, INVESTED_IN, CONTROLS, HOLDS, and SUPPORTS. That is a deliberately small schema, and its smallness is the feature. Six node types and five edge types are enough to express the questions the business actually asks, and few enough that every extraction can be validated against them and every query can be written without guessing. Notice that two of the node types, Document and Extraction, are not business entities at all, they are provenance machinery, which we will come back to. The lesson is not to copy this schema, it is to design yours the way you would design a database schema, before you write a single row, with types chosen to match the queries you know you need to answer and nothing you do not.

Extraction is a pipeline, not a prompt

Here is the sentence that separates a demo from a production graph: extraction is a pipeline, not a prompt. The tutorial version of GraphRAG is one call to a model that says "extract the entities and relationships from this document," and it works well enough on a clean document to make a convincing demo. It falls apart on a real corpus, because a real corpus has scanned PDFs, mixed document types, tables that carry the actual facts, and claims that need to be checked against the source rather than trusted because the model asserted them. Production extraction is a sequence of stages, each doing one job it can be tested on.

On the platform above, the document-intelligence pipeline runs eight stages: parse, classify, evidence, extract, validate, score, orchestrate, and graph-write. Parse turns the raw file into clean structured text and tables. Classify decides what kind of document it is, because a private placement memorandum, a SAFE, a K-1, and a cap table each need different extraction logic. Evidence locates the specific spans that support each fact, so nothing gets extracted without a source. Extract pulls the typed entities and relationships against the schema. Validate checks them against the schema and against each other, so a cap table that does not sum to a hundred percent gets flagged rather than written. Score attaches a per-field confidence. Orchestrate resolves conflicts across documents and across the extractions already in the graph. And graph-write commits the result as typed nodes and edges. Each stage has a distinct failure mode, which is the reason to separate them: when the graph is wrong, you can point at the stage that produced the fault instead of re-running one giant prompt and hoping. The single-prompt version has exactly one failure mode, which is "the output was wrong," and no way to tell you why.

Resolve entities at write time, not read time

The most expensive construction mistake is deferring entity resolution to read time, so this one is worth being emphatic about. In any real corpus the same entity appears under many surface forms. Acme Corp, Acme Corporation, Acme, and Acme Holdings are one company wearing four names, and if all four land in the graph as separate nodes, every query about Acme silently retrieves a quarter of the truth. Entity resolution is the process of collapsing those surface forms into one canonical node with a stable ID, and the only correct place to do it is at write time, before the node lands, because once four duplicate nodes exist with edges hanging off each of them, untangling them is graph surgery, and until you do, every retrieval over-fetches or under-fetches and no retriever setting can fix it.

Doing it at write time is not one step, it is a small pipeline of its own: canonical-name lookup, embedding similarity to catch near-duplicates, and rule-based matchers on hard identifiers like a registration number when one exists. The hard half is not merging the names that match, it is catching the merges that should not happen, because Acme Holdings and Acme Defense can look nearly identical to an embedding while being legally distinct entities that must never be collapsed. We run a dedicated resolution pass and then an LLM-driven mislink detector that flags suspicious merges before they corrupt the graph, because a wrong merge is worse than a missed one: a missed merge splits an entity and loses some connections, but a wrong merge invents connections that were never real, and an invented connection is a false fact the graph will assert with total confidence. The full mechanics of collapsing many names into one identity are their own deep-dive on entity resolution, and it is the single highest-leverage stage in the whole construction pipeline.

Provenance is a first-class citizen, not a footnote

Remember the two node types that were not business entities, Document and Extraction. They exist so that provenance is part of the graph, not an afterthought bolted onto it. Every fact in a production graph should be able to answer the question "how do you know that," and the only way to guarantee it is to make the answer a structural property of the graph rather than a hope. On our platform, every extracted fact is a node connected by a SUPPORTS edge back to the Document it came from, carrying the source span and a per-field confidence. So a claim in the graph is never a bare assertion. It is a claim with a pointer to the exact passage in the exact document that supports it, and a number saying how sure the extractor was.

Provenance is not documentation, it is the load-bearing structure that makes three other things possible. It makes grounding possible, because the language model at answer time can be required to cite the source span behind every claim, which is the same discipline that anti-hallucination grounding against a domain vocabulary relies on: a model that must cite cannot wander. It makes audit possible, because when a user asks why the system believes something, the answer walks from claim to span to source document, which for legal, financial, and healthcare work is not a nicety, it is the difference between a system that can be deployed and one that cannot. And it makes repair possible, because when a source document changes, you can find every claim that depended on it and re-verify exactly those, instead of rebuilding the graph. A graph without provenance is a pile of assertions you have to trust. A graph with provenance is a set of claims you can check, and the difference is decided entirely at construction.

A vector index that is wrong costs you one chunk. A graph that is wrong costs you every query that walks through the wrong edge. That is why construction, not retrieval, is where GraphRAG is won or lost.

Validate at insertion, or pay for it at retrieval

Every stage above produces something that can be wrong, so construction needs a gate, a validation step that runs before anything is committed and rejects what does not belong. Without it, the graph accumulates the specific failures that make graphs lie: orphan nodes that were extracted but never connected to anything, duplicate identities that entity resolution missed, edges with no type, edges with no provenance, and low-confidence claims written as if they were certain. None of these announce themselves. They sit in the graph looking exactly like valid data, and they surface later as wrong answers that you will, predictably, try to fix in retrieval.

The gate is cheap to build and expensive to skip. It checks each node against the schema, each edge for a valid type and a valid pair of endpoints, each fact for a provenance link and a confidence above a threshold, and it either fixes, quarantines, or rejects anything that fails. The claims it quarantines are not thrown away, they are held for a human or a second pass, because a low-confidence extraction is a signal, not noise. Skipping this gate does not save you the work, it moves the work downstream and multiplies it, because a bad claim caught at insertion is one row to fix, while the same bad claim discovered in production is a wrong answer, an investigation, and a repair to a graph that has since grown edges on top of the fault. This is the construction-time version of the same lesson graph-rot monitoring teaches at run time: it is far cheaper to keep bad data out than to find it once it is in.

The construction test: can you answer a three-hop query?

There is a simple test for whether your construction is any good, and it does not involve looking at the graph at all. Ask it a real multi-hop question and see if the answer falls out. On the investment platform, a question the business genuinely asks is "which vehicle holds our position in this company, and who controls that company." Answering it walks three hops through the schema: from the Vehicle, along HOLDS to the Investment, along INVESTED_IN to the Company, and back along CONTROLS from the Person. If construction was done right, that traversal is trivial, because every node is canonical, every edge is typed and true, and the path simply exists. The retriever does almost nothing clever; it walks a path the graph already encodes.

Now watch the same query fail on a badly constructed graph, and notice that the failure is always upstream. If the company was split into three duplicate nodes, the position hangs off one and the controlling person off another, and the path is broken. If the CONTROLS edge was mistyped as a generic relationship, the traversal cannot find it. If a wrong merge fused two companies, the query returns a controller for the wrong one, confidently. In every case the retriever did its job perfectly and returned a wrong answer, because the answer was made wrong at construction. This is the whole thesis in one query: a graph built right makes multi-hop retrieval easy, and a graph built wrong makes it impossible, and the retriever is never the variable. When your multi-hop answers are bad, run this test, and let it point you upstream to the stage that broke.

When a knowledge graph is the wrong tool

The honest close is that the best-constructed graph is the one you did not build because you did not need it. A graph earns its construction cost only when your value is in relationships, multi-hop questions, entity resolution across sources, and provenance you can audit. When the hard problem is something else, a graph is expensive overhead that adds failure modes without adding answers. We have built vector-only systems on purpose for exactly this reason. One is a K-12 education retrieval system with no graph at all, five hundred and seventy-seven chunks across twenty-eight documents in a single vector index, because its hard problem was vocabulary and phrasing, not relationships between entities, and a graph would have been machinery in service of a question nobody was asking. The decision of whether the vector database is even the wrong tool yet comes before any of the construction discipline in this piece.

So the first construction question is not "how do I build the graph," it is "do I need one," and the answer is no more often than graph enthusiasm admits. But when the answer is yes, when the relationships are the value and the questions are multi-hop and the provenance has to hold up in an audit, the graph is the system, and its quality is decided in the pipeline that builds it, long before any user types a query. Design the schema first, run extraction as staged pipeline, resolve entities at write time, make provenance structural, and gate every insertion, because once the graph is built, retrieval can only read back what you wrote. You cannot retrieve your way out of a bad graph.

We build production GraphRAG systems for document-heavy work in legal, finance, and insurance, where the graph carries provenance that has to survive an audit and a wrong edge is not a bad search result, it is a claim about the world that a filing might depend on. The thing that keeps those systems trustworthy is not a cleverer retriever, it is construction: a schema designed before extraction, a staged extraction pipeline with a validation gate, entity resolution at write time, and provenance on every claim. If that is the kind of system you are trying to build, talk to an engineer or see how we work.