An Agent That Saves Everything Remembers Nothing

TL;DR: Telling your agent to remember everything does not give it a memory. It gives it a landfill: stale facts, contradictions it never reconciled, and a retrieval step that now drags back noise and buries the one detail that mattered. Storing everything and storing nothing fail the same way. The work is the write path, the step between capturing history and reading it back, and it has four operations: extract the durable facts, summarize only the narrative you cannot keep as facts, reconcile contradictions with a timestamp instead of overwriting, and let stale memory decay. Add reflection to turn a pile of observations into the higher-level conclusion they imply, and run the whole pass in the background so it costs nothing when the user is waiting. Do this and a small model stays sharp and consistent across months of interaction. Skip it and you have built an expensive landfill with excellent search.

An agent that has been told to remember everything is not building a memory. It is building a landfill. Two months into production it is holding ten thousand stored turns, a third of them stale, a handful of them flatly contradicting each other, and every retrieval now hauls back a pile of near-duplicates that buries the single fact the current question needed. The reflex, after the last three posts, is to store more and retrieve harder. But the system was never short on storage and the retriever was never the problem. What was missing is a decision, made on the way in, about what was worth keeping and in what form.

This is the fifth post in our series on agent memory. The first established that a context window is not a memory. The second compared the tools for building the store that is. The third showed that getting the right facts out of that store is a ranking problem. The fourth showed how to hold a working set in a finite window. Those four posts covered storing memory and reading it back. This one is about the step in between, the one most teams skip: the write path, where raw history becomes durable memory, or fails to. Get it wrong and the best retrieval in the world is just ranking garbage.

Memory is not what your agent stored. It is what your agent kept, in a form it can still use. Everything else is history you are paying to carry, search, and trip over.

Storing everything is not remembering

The most common memory architecture in production is also the most naive: save the whole transcript, embed it, and run retrieval over it later. It demos fine and it degrades on a schedule, for three reasons that all get worse as the log grows.

Precision falls. The more raw turns you store, the more near-duplicates and loosely-relevant chunks every query has to compete with, so the genuinely relevant fact is harder to surface, not easier. You made the haystack bigger and left the needle the same size.

Contradictions accumulate. On turn four the user said the budget ceiling was one number. On turn nine hundred they said it was another. A raw log keeps both, side by side, with no signal about which is current, so retrieval can and will hand the model the stale one at exactly the wrong moment.

Cost climbs for nothing. You are paying to embed, store, and search material that no one will ever read, and that bill grows for as long as the agent runs.

There is peer-reviewed weight behind the intuition. LongMemEval, published at ICLR in 2025, tested assistants on sustaining memory across long interactions and found accuracy dropping by roughly thirty percent as the history stretched out, with structured memory beating the approach of reading raw history back into context. Raw history, it turns out, is not memory at all. It is the raw material that memory is made from, and consolidation is the making. An agent that saves everything has done the first ten percent of the job and called it done.

Memory has a write path, and it is where the work is

The last two posts were the read path. Retrieval decides what to bring in; working memory decides what to keep in the window. Both of them quietly assume the store already holds clean, current, well-shaped memory to bring in and keep. The write path is what makes that assumption true, and it is where most of the engineering actually lives.

It helps to borrow the vocabulary from a peer-reviewed source. CoALA, published in the journal TMLR in 2024, decomposes an agent's memory into four kinds: working memory, the active state on this turn; episodic memory, the time-stamped record of what happened; semantic memory, the distilled facts about what is true; and procedural memory, the skills and routines the agent can run. The crucial move for our purposes is the transform from episodic to semantic. Episodic memory is "on Tuesday the user said the deadline moved to March." Semantic memory is "the deadline is March 3." Consolidation is the verb that gets you from the first to the second, and it is precisely the step a raw-log architecture never performs.

This is why retrieving harder does not save a badly written store. If what you stored is raw episodic sludge, the retriever is ranking sludge, however good its ranking is. Most teams build the read path because it is visible, the part the user touches, and skip the write path because it runs out of sight. Then they spend months tuning retrieval against a store that was never consolidated in the first place. The order is backwards. Decide what to write, and in what form, before you worry about how to read it.

The four operations: extract, summarize, reconcile, decay

The write path is not one action. It is four, and the discipline is doing all four deliberately instead of dumping turns into a vector index and hoping.

Extract the durable facts. From each turn, pull the discrete, queryable atoms: the entities, preferences, decisions, and constraints. Not a paragraph about the turn, atoms. "Prefers email over phone." "Budget ceiling is X." "Chose vendor Y on this date." This is what makes retrieval precise later, because you are storing answers, not transcripts. Production memory libraries do exactly this with an extraction pass: Mem0, for instance, reads each new exchange, proposes the facts worth keeping, and then decides for each one whether it is new, an update to something already stored, or already known and safe to drop. The output is a clean set of facts, not a copy of the conversation.

Summarize only what you cannot keep as facts. Some material is genuinely narrative: the arc of a discussion, the reasoning that led to a decision, the shape of a long document. That you compress into a short gist, knowing the compression is lossy. Treat summarization as the tool of last resort, because it is exactly where detail goes to die. OpenAI's 2021 work on recursively summarizing books, a technical report rather than a peer-reviewed paper, is a useful caution: even with heavy effort, only about five percent of its full-length summaries reached human quality, because deep compression sheds the specifics. So summarize the narrative, never the fact you could have extracted.

Reconcile contradictions instead of overwriting. When a new fact contradicts a stored one, you have three poor options and one good one. You can keep both, and let retrieval surface a contradiction. You can silently overwrite, and lose the history and the audit trail. You can ignore the conflict, which is keeping both with extra steps. Or you can reconcile: mark the old fact superseded as of a timestamp and record the new one as current, so the present truth is unambiguous and the past stays intact. Temporal knowledge-graph systems are built around this move. Graphiti and Zep use a bi-temporal model that invalidates a fact as of a point in time rather than deleting it, so the graph knows both what is true now and what was true then. Mem0's update-and-delete step is the lighter-weight version of the same idea. Either way, a memory that changes should leave a trail, not a blank.

Let stale memory decay. Not every fact earns permanent residence. A preference the user has since reversed, a decision that was later changed, a detail about a ticket that closed months ago: left in the store, these rot it into the same landfill we started with, crowding the live facts with dead ones. So memory needs an expiry policy: down-weight or retire facts that have not been touched, that have been superseded, or that were always low-value. This is the same discipline as keeping a knowledge graph fresh and fighting the graph rot we covered in the first pillar. Forgetting, done on purpose and with a policy, is a feature.

Reflection turns observations into memory worth keeping

Extraction captures facts that were stated. The highest-value write-path move, and the one almost no production system implements, captures the facts that were never stated but follow from many that were. That move is reflection.

Stanford's Generative Agents, published at UIST in 2023, built it explicitly. Every memory gets an importance score from one to ten. When the cumulative importance of recent events crosses a threshold, one hundred and fifty in their implementation, the agent pauses and reflects: it asks what the recent memories imply, and writes the higher-level conclusions back into memory as new entries, which can themselves later be reflected on. A scatter of observations, "asked about SOC 2," "asked about data residency," "asked about audit logging," consolidates into one durable inference: "this user is security-driven." In their controlled ablation, agents with reflection scored measurably higher on a believability metric than agents without it, though that metric was human-rated believability inside a simulation, so read it as directional rather than as a production memory benchmark. The mechanism is the point: reflection is the episodic-to-semantic transform CoALA names, made concrete.

It is worth separating this from a different thing that also gets called reflection, because an expert will notice if you blur them. Reflexion, published at NeurIPS in 2023, has an agent reflect on its own task failures, success or failure on a problem, and store the lesson to do better on the next attempt, reaching ninety-one percent on a coding benchmark against a strong baseline. That is task-feedback reflection: learning from outcomes. It is a cousin of memory consolidation, not the same thing. Both write text memory out of experience, but only the Generative-Agents kind is about consolidating what the agent has observed into what it should remember. When someone says their agent "reflects," it is worth asking which of the two they mean, because they solve different problems.

Summarization is lossy, and structure beats it

The tempting shortcut for the write path is "every N turns, summarize the conversation and replace it with the summary." It is better than keeping everything raw and worse than it looks, and the reason matters for how you build.

Summaries smooth away exactly what careful work later depends on: the precise quantity, the specific identifier, the exact wording of a constraint. The gist survives and the detail does not, which is fine until a downstream turn needs the detail and finds only the gist. Two peer-reviewed benchmarks point the same direction here. LongMemEval, again, found that structuring memory, extracting facts and indexing them, improved both recall and the final answers compared with dumping raw history into a long context window. LoCoMo, published at ACL in 2024 on very long conversations, its widely used public subset being ten conversations that each run to several hundred turns, and that is the subset I am citing, found that long-context models and plain retrieval both still trail humans by a wide margin on sustained memory.

One caveat belongs here, stated plainly because the field rarely does. The public benchmarks for agent memory are young and their headline numbers are contested. The widely-cited memory leaderboards are mostly run by the memory vendors themselves, independent audits have found real errors in their answer keys and automated judges, and in several of those vendors' own papers a plain full-context baseline beats their memory system on raw accuracy. That last point is not the embarrassment it looks like. Stuffing the entire history into the window can be accurate, it simply does not scale, for the overflow and quadratic-cost reasons the previous post laid out. So the honest case for consolidation is not that it tops a leaderboard. It is that it holds accuracy roughly steady while cutting the tokens, latency, and cost of every turn, and keeps a long-running agent consistent instead of letting it drown in its own history. Build it for efficiency and consistency, and treat any single accuracy number, a vendor's or ours, with suspicion.

Read together, the lesson is not "summarize harder." It is that raw full history is fragile and naive summarization is not the cure; structured extraction is. So when you do summarize, summarize the narrative and extract the facts, and never let a summary become the only home of a detail you will later need exactly. The strongest systems do both: a few atoms pulled out precisely, plus a short gist for the connective tissue. Summarization alone is how you build a memory that sounds right and gets the numbers wrong.

Consolidation costs compute, so do it off the hot path

Every operation above is an extra model call. Extraction reads the turn. Reflection reasons over many memories. Reconciliation compares the new fact against what is stored. Do all of that inline, on the turn, and you have added latency to every single response, for memory the user will not need until much later, if at all. That is the tax that makes teams skip the write path entirely.

The fix is timing, and the research has a name for it. Sleep-time compute, an arXiv preprint from 2025 and so not yet peer-reviewed, showed that moving the heavy inference offline, doing it between queries rather than during them, cut the compute needed at answer-time by roughly five times for the same accuracy. Its tasks were reasoning problems rather than chat memory, so the exact multiple belongs to that setting, but the principle transfers cleanly: consolidate in the background. After a session ends, during idle time, or asynchronously right after a turn while the user reads the reply, run the extract, reflect, and reconcile pass and write the results to the store. The hot path stays fast, and the memory is ready before the next turn asks for it.

There is an old idea underneath this, and it is worth naming honestly as an analogy and nothing more. Brains consolidate memory offline: a fast-learning system, the hippocampus, captures the day's episodes, and a slow system, the neocortex, is taught them over time through replay, much of it during sleep, a model neuroscientists have held since the 1990s. "Sleep-time compute" borrows that name on purpose. It is a metaphor, not a mechanism. No summarizer literally sleeps, and nothing here implements biology. But the shape the metaphor points at, a fast path that captures and a slow background path that consolidates, is exactly the right architecture for agent memory, which is probably why the same shape keeps getting reinvented.

What this looks like when we build it

The abstract version is extract, summarize, reconcile, decay, in the background. Here is the concrete version, from systems we have shipped, anonymized.

On a wealth-management platform we built, the write path is the product. Documents arrive and an eight-stage pipeline extracts the entities inside them into a knowledge graph in Neo4j, six entity types and five relationship types, with a confidence score from zero to one on every fact. That is extraction, not summarization: the documents are the episodic record, and the graph is the consolidated semantic memory the agents actually use. When a new document changes a fact an earlier one established, the system reconciles rather than overwrites, and the graph is kept current the same way we described in the pillar on keeping a knowledge graph fresh, because a consolidated store that is allowed to rot is no better than the landfill we started with. The agents answering questions never touch the raw documents. They read the consolidated graph, which is why their answers stay precise as the document pile grows instead of degrading with it.

On Paralegent, our multi-agent legal-analysis system, the agents do not pass raw transcripts to each other. Its twenty-three agents, twelve scorers and eleven analysts, write structured results into a shared scores table, consolidated state that any agent can read the relevant row of without re-reading anyone's full output. A router decides which agents and which facts a given document actually needs and prunes the rest, so the system writes and reads compact, structured memory instead of broadcasting full context to everyone, and moving that shared state out of the prompts and into the table, with the pruning, is a large part of why model calls dropped by roughly seventy-five percent. The shared table is consolidation made physical: not a summary of what the agents discussed, but the structured facts they concluded.

Neither system stays coherent at scale because of a clever retriever or a longer window. Both stay coherent because the memory the agents read was written deliberately: extracted into facts, reconciled when it changed, and kept current, never a raw log they have to wade through.

How to tell consolidation is your bottleneck

A handful of symptoms point at the write path specifically, rather than at the model, the retriever, or the window.

The agent contradicts itself over time, asserting one thing this week and the opposite next, because both facts sit in the store and nothing ever reconciled them. Retrieval quality degrades as the store grows even though you have not changed the retriever, because the store is filling with stale and duplicate facts that crowd out the live one. The agent confidently "remembers" things that are no longer true, a preference the user reversed, a decision that was changed, because nothing decays. Memory grows without bound and the bill for storing, embedding, and searching it climbs with it. And answers come out vague where they should be exact, because the only surviving record of a detail is a summary that smoothed it away.

When that is the pattern, retrieving harder will not help, because the problem is upstream of retrieval. The fix is the write path: extract durable facts instead of storing raw turns, summarize only the narrative and never the specifics, reconcile contradictions with a timestamp instead of letting them pile up, let stale memory decay on a policy, and run the whole pass in the background so it costs nothing when the user is waiting.

The thread through this series closes here. A context window is not a memory. Retrieval decides what is worth bringing in. Working memory decides what is worth keeping in front of the model. And consolidation, the step most teams skip, decides what was ever worth remembering in the first place. Build all four and your agent gets sharper the longer it runs, because its memory gets denser and cleaner instead of larger and noisier. Skip the write path and you have built a very expensive landfill with excellent search.

Is your agent's memory getting noisier the longer it runs, contradicting itself or going vague on details it should have nailed? That is almost always a write-path problem, not a retriever or a model problem, and building the consolidation layer, extract, reconcile, decay, in the background, is work we do every week. Book a 15-minute call and we will tell you honestly whether your bottleneck is the write path, the retrieval, or the store. We work US business hours.