Do You Actually Need a Knowledge Graph?

Most teams decide they need a knowledge graph for the wrong reason. They read that GraphRAG beats plain retrieval, they see a competitor mention a graph, and they conclude that a graph is the upgrade their AI has been missing. Then they spend three months building one, and their agent answers the same questions it answered before, only slower and at higher cost.

The opposite mistake is just as common and more expensive. A team that genuinely needs a graph builds a pile of embeddings instead, ships it, and spends the next year confused about why their agent keeps giving confident answers that fall apart the moment a question requires connecting two facts.

A knowledge graph is not an upgrade. It is a different tool for a different question. We have built both kinds of system, a graph-backed platform for a family office and a vector-only retrieval system for an education company, and the most useful thing we can tell you is how we decided which was which. This is the sixth post in our series on graph rot, and it is the one to read before you commit a single sprint to building one.

Reach for a graph when the answer lives in the connections, not the content.

What does a knowledge graph give you that a vector database does not?

It gives you relationships as first-class facts, instead of relationships you have to hope the model infers from nearby text.

A vector database stores your documents as embeddings and finds the chunks that are semantically closest to a question. That is retrieval, and for a large share of AI systems it is exactly the right tool. Ask it "what does our refund policy say about damaged goods" and it will find the passage, hand it to the model, and the model will answer. A knowledge graph stores entities and the explicit edges between them, so you can traverse from one thing to the next instead of matching text.

Here is what that difference looks like in a real system. On a family-office investment platform we built, the graph has six node types, Company, Investment, Document, Person, Vehicle, and Extraction, joined by five explicit edge types: a Vehicle HOLDS an Investment, an Investment is INVESTED_IN a Company, a Person CONTROLS a Company, a Document is EXTRACTED_FROM a Company, and a Document SUPPORTS an Extraction. Those edges are the product. They let the system answer a question no vector store can: "which vehicle holds our position in this company, and who controls the company." Answering that walks three hops, from vehicle to investment to company to the person who controls it, and every hop is a stored fact rather than a guess from nearby text. A vector store can find the documents that mention the company. It cannot walk the chain, because it never stored the chain.

The clean way to hold the difference: a vector database is the right tool when the answer is in the content of one place, and a graph is the right tool when the answer is in the connections between many places.

When is a vector database enough?

When your hardest question is a single-hop lookup over the content of your documents, you do not need a graph, and building one is a tax you will pay forever.

We mean this from experience, not theory. One of the production systems we are proudest of has no graph at all. It is a retrieval system over a focused library of teaching material, 28 documents and roughly 1.37 million characters, cut into 577 chunks that live in a single vector index. It runs hybrid search, dense embeddings for meaning and sparse keyword matching for exact terms, fused with reciprocal rank fusion, because the hard problem in that domain was vocabulary, not relationships. The corpus uses specific named techniques that a purely semantic search would paraphrase away, so exact-term matching had to carry equal weight. The relationships in that material never needed traversing, so we did not build a graph. We scoped results with metadata filters on the flat index instead. A graph would have been expensive decoration. The system is faster, cheaper, and far easier to keep current without one.

Here is the honest disqualifier, and we lead with it because the cheapest knowledge graph is the one you correctly decide not to build. You probably do not need a graph if your real questions look like that education system: semantic search over a document set, a support bot that answers from a policy library, summarizing or classifying text, "find me passages similar to this one." Every one of those is single-hop, content-bound retrieval, and a well-built vector pipeline will serve it faster, cheaper, and with far less to maintain than a graph.

And the hardest disqualifier to hear: you should not build a graph you cannot keep fresh. A graph that ingests new documents picks up new errors on every load, which is the entire subject of keeping a graph fresh. If you do not have the capacity to maintain it, a graph does not stay an asset. It rots into a liability that produces confident, wrong answers. A vector index that goes a little stale degrades quietly. A graph that goes stale lies with structure behind it.

When do you actually need a graph?

When answering your most important question requires connecting facts that live in different places, treating different names as the same thing, or proving how you reached an answer. There are three signals, and one is usually enough.

The first is multi-hop questions. The vehicle-to-investment-to-company-to-person chain above is the clearest example: no amount of semantic similarity assembles that reliably, because similarity does not know that one fact leads to the next. A vector store can find documents that mention a director. It cannot answer "who controls the company our fund is invested in through this vehicle," because that answer is a path, and a vector store does not store paths. If your highest-value question is a path, you have crossed into graph territory.

The second is identity that has to be enforced across messy sources. On that same platform, one company arrives as "Acme Corp" in a SAFE, "Acme Corporation" in a cap table, and "ACME Inc." in a K-1, and every answer is wrong unless those are treated as one node. A dedicated cross-document linker, backed by LLM disambiguation and a mislink-detection pass that flags edges that should not exist, collapses those into a single entity. That is entity resolution, and it is structurally a graph problem: a vector store has no place to record "these are the same," only a fuzzy sense that two chunks sound alike.

The third is provenance you can defend. In finance and legal work, "the agent said so" is not an acceptable answer. The extraction pipeline on that platform attaches, to every field it pulls, the verbatim passage it came from and a confidence score, and the graph stores that as a first-class edge, the Document SUPPORTS the Extraction. So the system returns not just "this company is valued at this figure" but the exact sentence in the exact document that says so, and the semantic-search layer does the same on the unstructured side, returning answers with citations. Provenance is the deliverable, and a graph is how you store it so the path back to the source is always one edge away. This is why our legal and finance agent work leans on graphs.

So how do you actually decide?

You run your hardest real question through one decision path, and the path tells you which tool you are actually buying.

Take the single most valuable question you want your AI to answer, the one that justifies the project, and walk it down the tree. Does answering it require connecting facts across different documents or sources, resolving the same entity under different names, or returning a traceable path. If the honest answer to all three is no, stop. You want a vector database and a good retrieval pipeline, and you will ship faster for admitting it. If the answer to any of them is yes, you are in graph territory, and the next question is not whether to build a graph but which graph to build.

The mistake to avoid is running an average question down the tree. Average questions are almost always single-hop, so they will tell you to skip the graph even when your highest-value question needs one. The education system answered thousands of single-hop questions and never wanted a graph. The investment platform lived or died on three-hop questions and could not work without one. Decide on the hardest question that matters, not the typical one.

What does a real one look like?

A system that genuinely needs a graph rarely has only a graph. It has a graph and a vector index, each doing the job it is good at.

The investment platform is the shape above. Documents arrive, an extraction pipeline turns them into entities, and the result lands in two stores at once. A managed Neo4j graph holds the entities and their relationships and provenance. A separate semantic search index holds the unstructured passages and returns answers with citations. The agents query both, the graph for connected questions and the search index for "what does this document say." Neither store does the other's job, and the architecture is honest about that instead of forcing one tool to fake the other. When people say GraphRAG, this hybrid is usually what they actually mean: a graph for structure, retrieval for content, combined.

Should you build one or buy one?

Buy or adopt off-the-shelf when your domain is generic and your relationships are standard. Build custom when your identity problem is hard, your ontology is specific, or your provenance has to hold up under scrutiny.

The buy path is real and you should take it when you can. On the investment platform, the infrastructure was bought, not built: a managed Neo4j service for the graph, a managed search engine for semantic retrieval, an off-the-shelf graph-orchestration library to wire them together. If your entities are common, your relationships are obvious, and nobody is going to audit the result, adopt the existing stack and move on. Building from scratch what you could have configured is its own kind of waste.

What we built was the part no framework ships. An eight-stage extraction pipeline that turns a messy private placement memorandum, SAFE, or cap table into validated, scored, sourced graph nodes, using a strong model for structured extraction. The cross-document entity resolution that decides eleven names are one company. The reconciler that flags when an ownership figure extracted from a document disagrees with the cap table. That is the work our data engineering tends to land in, and it maps onto the discipline of scoring a graph before you trust it. The database was a purchase. The correctness was the build.

A blunt rule of thumb. If a wrong answer is an inconvenience, buy. If a wrong answer is a liability, the extraction, the resolution, and the validation are the hard part, and that is the part worth building carefully.

What does a knowledge graph cost you after you ship it?

The cost of a knowledge graph is not building it. It is keeping it true.

This is the line item teams forget, and it is the one that decides whether the graph was worth it. On the investment platform, keeping the graph current is a system in its own right. New documents arrive through signed Drive webhooks and a Gmail push subscription, so the graph updates within minutes of a file landing. A polling job runs every ten minutes as a safety net for anything the webhooks miss. And here is the cost nobody budgets for: those webhook subscriptions expire, so a scheduled job renews them on its own clock, the Drive channels every twelve hours and the Gmail watch every day. Every new document is also a fresh chance to introduce a duplicate entity, a mislinked edge, or a stale fact, which is the whole reason graph rot is the name of this series, so the resolution and validation run on every ingest, not just at launch.

Now compare the vector system. Its corpus is static and curated, so keeping it current is a delete-and-rebuild that re-ingests all 577 chunks in about nine seconds. Same requirement, keep it fresh, and two completely different cost structures. A graph attached to live sources needs continuous governance. A vector index over a slow-moving corpus barely needs attention. Which one you are signing up for should decide the build, because the freshness machinery, not the database license, is the real line item.

So the buyer's question is not "can we build a graph." Almost anyone can stand up a graph that looks done. The question is "can we keep one true," because a graph you cannot maintain is more dangerous than the vector store you replaced. Count the maintenance before you count the benefit. If the maintenance math does not work, the honest answer is the vector database, and we would rather tell you that now than after the build.

What did building both teach us?

The first lesson is the one this whole post is built on: we have shipped one of each, on purpose. A graph-backed platform where the value is in how vehicles, investments, companies, and people connect, and a vector-only system where the value is in finding the right passage in a focused library. The teams that get the most from a knowledge graph are the ones who, like the investment platform, had a real multi-hop question, a real identity problem, or a real provenance requirement, and could name it before a line of code was written. The teams that struggle are the ones who built a graph to keep up, then went looking for questions it could justify.

The second lesson is that the line between the two tools is not as sharp as the vendors make it sound, and the best systems are hybrids. A vector layer for content-bound retrieval, a graph layer for the connected questions, each doing the job it is actually good at. The decision is rarely "graph or not." It is "where does the value actually live, in the content or in the connections," and the honest answer is often "both, in different places," which is exactly what our enterprise retrieval work usually is.

If you can name your hardest question, you can answer the graph question yourself. If you cannot, that is the real problem to solve first, and no graph will solve it for you.

We build and fix knowledge graphs for AI systems, and we will tell you when you do not need one. If you are weighing a graph against a simpler retrieval system and want a straight answer, book a 15-minute call.