Memory Is More Than Context

The call that forgets

Start with the primitive, because everything else is scaffolding built on top of it. A call to a language model takes a sequence of tokens and returns a sequence of tokens:

\[ y_t \;=\; \mathrm{LLM}_\theta\!\left(x_t\right) \]

and then it is over. The weights \(\theta\) are unchanged. No hidden variable survives to the next call. Whatever the KV cache held is either evicted or, at best, reused as a prefix cache for latency — a performance optimisation, not a state channel. If something that happened during call \(t\) is going to matter during call \(t+1\), some program outside the model has to put it into \(x_{t+1}\) as tokens.

That program is the agent. This is the entire subject of this article: the agent is the thing that decides what goes into the next \(x\).

The analogy I keep coming back to is a brilliant consultant with total anterograde amnesia. Every morning they walk into the room having forgotten everything that ever happened, read whatever briefing folder is on the desk, give you genuinely excellent advice, and forget again. Their competence is real and their training is intact. What they lack is continuity. And the quality of their advice is bounded not by their intelligence but by whoever assembles the folder.

So there are exactly three places knowledge can live in such a system, and it is worth being pedantic about the distinction, because most confused arguments about agent memory come from conflating them.

Parametric memory

Knowledge compressed into \(\theta\) during pretraining and post-training. It is enormous, cheap at inference (you do not pay tokens for it), generalises well, and is essentially impossible to edit precisely or to attribute. The model knows that Paris is in France without you saying so; it also cannot tell you when it learned that, or forget it on request.

Contextual memory

The tokens \(x_{1:n}\) present in the current forward pass. Perfectly fresh, perfectly attributable, editable at will, and gone the moment the call returns. It also costs money and attention, which is the whole reason this article exists.

External persistent memory

Anything stored outside the model — files, rows, vectors, graph edges — that some retrieval step can pull back into \(x\) later. Durable, inspectable, editable, deletable. Useless unless something puts the right pieces into context at the right moment.

The last one is where all the interesting engineering is, and it exists purely to bridge the other two. An agent's long-term behaviour is determined by the policy that moves information from the external store into the context window, and by the policy that decides what enters the external store in the first place.

Context is not memory

Two claims worth nailing to the door before we go further, because a great deal of engineering effort is wasted by people who believe their negations.

Context is the working set for one decision. Memory is the store the working set is drawn from. Conflating them is like conflating your desk with your filing cabinet. A desk is not a small filing cabinet; it is a different object with a different job. You do not solve “where do I keep my records” by buying a bigger desk, and you do not solve “what should I be looking at right now” by dumping the cabinet onto the desk.

This is the one that costs teams the most time. Context windows have grown by orders of magnitude, and every time they grow someone announces that memory is now a solved problem: just put the whole history in. It does not work, for four separate reasons that we will take apart in §5 — but the shortest version is that a long context is a capacity claim, not a utilisation claim. Being able to fit a million tokens tells you nothing about whether the model can reliably use the fact sitting at token 480,000. And even where it can, you are paying prefill compute and per-token cost on the other 999,999.

There is also a structural argument that has nothing to do with model limitations. An agent that runs for months produces an unbounded observation stream. No fixed window, however large, holds an unbounded stream. Sooner or later you are choosing what to keep. The only question is whether you choose deliberately or let a truncation rule choose for you.

What the model actually sees

Let us be concrete about the object we are engineering. At agent step \(t\), the tokens fed to the model are a concatenation of several different kinds of thing, each arriving from a different source and each governed by a different policy:

\[ C_t \;=\; \big[\, S,\; T_t,\; M_t,\; A_t,\; P_t,\; H_t,\; O_t,\; U \,\big] \]

• \(S\) — system and developer instructions: the agent's standing policy.
• \(T_t\) — tool definitions currently exposed. Every unused tool schema is rent paid on tokens. See Teaching LLMs to Use Tools for how these are interpreted.
• \(M_t\) — memories retrieved from the persistent store.
• \(A_t\) — artifacts: file contents, documents, retrieved passages.
• \(P_t\) — the plan and task state: what we are trying to do, what is done, what remains.
• \(H_t\) — recent conversation turns.
• \(O_t\) — recent environment observations and tool results.
• \(U\) — the current user request.

Only \(U\) is given to you. Everything else is a choice. That is why context engineering is a better name than “prompt engineering” for what building agents actually involves — and it is the framing Anthropic uses in its own guidance on the subject: curating and maintaining the optimal set of tokens during inference, not just writing better instructions. The operative question at each step is not what should I say to the model but:

Note the two halves of that sentence pull in opposite directions, and both matter. “Sufficient” is a recall constraint: leave out the API key format and the call fails. “Smallest” is a precision constraint: include forty irrelevant memories and the model's attention is spread across noise. Every real system lives on the tension between them, and the figure below is that tension made visible.

Token counts in this figure are representative of a mid-sized coding agent step; they are illustrative rather than measurements from a specific deployed system.

The agent as a bounded-state controller

Here is the framing I find most clarifying, and it is worth writing down formally because it makes several later results fall out for free.

The agent observes an unbounded stream \(o_1, o_2, \ldots\). It maintains a persistent memory state \(M_t\) via a write map, constructs a bounded context via a read map, and acts:

\[ M_t \;=\; W(M_{t-1},\, o_t,\, s_t), \qquad C_t \;=\; R(M_t,\, q_t) \ \ \text{with}\ \ |C_t| \le B, \qquad a_t \;=\; \mathrm{LLM}_\theta(C_t) \]

Read that again with the constraint in view. \(|C_t| \le B\) is a hard channel capacity. The full history is unbounded. Therefore every agent with memory is a lossy compressor of its own history into a fixed-width channel, and the pair \((W, R)\) is the codec. Designing agent memory is designing a codec whose distortion measure is “how much worse are the decisions.”

This immediately tells you three things. First, there is no free lunch: any \(W\) that discards information is optimal only relative to some distribution over future queries. Second, \(W\) and \(R\) must be co-designed — a write format that is easy to store but impossible to index is worthless. Third, the interesting failures are compression artifacts: things that were true, got summarised, and came back subtly wrong.

The allocation problem

Once you accept that \(|C_t| \le B\), context construction stops being a matter of taste and becomes an optimisation problem. Let the candidate pool be \(\Omega = \{c_1, \ldots, c_N\}\): every memory, every file, every tool schema, every past turn that could go in. Each candidate has a cost in tokens and some usefulness for the decision at hand:

\[ \ell_i \;=\; \mathrm{tokens}(c_i), \qquad u_i \;=\; U(c_i \mid q, s_t) \]

and we want

\[ \max_{C \subseteq \Omega}\; U(C \mid q, s_t) \qquad \text{subject to} \qquad \sum_{c_i \in C} \ell_i \;\le\; B. \]

If \(U\) were additive — \(U(C) = \sum_{c_i \in C} u_i\) — this would be a 0/1 knapsack, and you would sort by density \(u_i / \ell_i\) and fill greedily. That is already a useful mental model, and it explains a real phenomenon: a 30-token fact that pins down the answer beats a 3,000-token document that merely contains it. Density, not relevance, is the currency. A long document with the answer buried in it has terrible density.

Why it is harder than knapsack

Utility is not additive. This is the crux:

\[ U(\{c_i, c_j\}) \;\ne\; U(\{c_i\}) + U(\{c_j\}) \]

and the deviation goes in both directions. Two items can be redundant (the same fact stated twice adds nothing: the marginal gain of the second is zero). They can be complementary (a schema plus an example of it is worth more than the sum: superadditive). They can be contradictory (“the user lives in Hyderabad” and “the user lives in Bangalore” together are worth less than either alone, because now the model must adjudicate and may pick wrong). And they can be distracting — a plausible but irrelevant memory that pulls the trajectory sideways.

The right primitive is therefore not the item's utility but its marginal utility given what is already in:

\[ \Delta(c \mid C) \;=\; U(C \cup \{c\}) - U(C) \]

and the greedy rule becomes: repeatedly add the candidate maximising \(\Delta(c \mid C) / \ell(c)\) while the budget allows. This single change absorbs all four interaction effects. A duplicate has \(\Delta \approx 0\) and is never selected. A contradiction has \(\Delta < 0\) and is actively excluded. A complement's \(\Delta\) rises once its partner is in, so it gets picked up on a later round.

If \(U\) happens to be monotone and submodular — diminishing returns, no negative interactions — density-greedy under a knapsack constraint has a well-known constant-factor guarantee. Real context utility is not submodular, because contradiction makes it non-monotone. So the guarantee does not transfer. But the greedy algorithm is still the right shape, and the standard practical approximation is maximal marginal relevance, which stands in for the interaction term with a redundancy penalty:

\[ \mathrm{MMR}(c \mid C) \;=\; \lambda\, \mathrm{rel}(c, q) \;-\; (1-\lambda)\, \max_{c' \in C} \mathrm{sim}(c, c') \]

That is a crude proxy — it models redundancy and nothing else — but it is cheap, and it captures the largest single effect in practice, which is near-duplicate memories crowding each other out of the budget.

A schematic lab. The nine candidate items carry hand-assigned utilities and token costs chosen to expose the three interaction effects (duplication, complementarity, contradiction); the resulting scores are illustrative of the mechanism, not measurements.

One more wrinkle, which we will need in the next section and which genuinely breaks the knapsack analogy: \(U\) depends on the ordering, not just the set. The same nine items arranged differently do not produce the same behaviour. So the true problem is sequence construction under a budget, and knapsack is the relaxation you solve first.

Accessible is not usable

Why not simply put everything in? There are four costs, and they are of genuinely different kinds — two are about resources, two are about the model.

1. Prefill compute

Attention is quadratic in sequence length; the rest of the forward pass is linear. Doubling the context more than doubles the prefill work. If the agent runs many steps, and each step re-reads a growing history, the total compute grows superlinearly in the number of steps. Prefix caching helps enormously when the prefix is stable — which is exactly why you put stable things (system prompt, tool schemas) first and volatile things last. The systems side of this is in The Serving Playbook; the mechanics of the quadratic term are in Attention, From First Principles.

2. Money and latency

Input tokens are billed and prefill takes wall-clock time. For a long-running agent, per-step context length multiplied by number of steps is often the dominant cost line, well above output tokens. An agent that re-reads 60k tokens of history on every one of 300 steps has read 18 million tokens to do one task.

3. Attention dilution

Softmax attention allocates a finite probability mass across the context. Adding irrelevant tokens does not create new attention; it divides the existing attention differently. This is the sense in which the model has an attention budget that you spend by including things. The practical signature is that adding a plausible-but-wrong memory is worse than adding nothing, because the model has to actively suppress it.

4. Position effects

The most-cited empirical result here is Lost in the Middle. Across multi-document question answering and key-value retrieval, Liu and colleagues found performance was often highest when the relevant information sat at the beginning or the end of the input context, and degraded significantly when the model had to reach for information in the middle — including for models explicitly built for long context. Performance also fell as the input grew longer, independent of position.

Take the consequence seriously. It means utilisation is a function of position:

\[ u_{\text{eff}}(c) \;=\; u(c) \cdot \phi\!\left(k, n\right) \]

where \(k\) is the item's position among \(n\) tokens and \(\phi\) is not flat. Two contexts containing exactly the same information can differ in effectiveness purely by arrangement. That is why the earlier “set selection” framing was a simplification, and it is why prompt-compression work treats position bias as a first-class target rather than as noise. LongLLMLingua frames long-context use as three coupled problems — computational cost, performance reduction, and position bias — and compresses the prompt specifically to raise the density and improve the placement of key information.

The U-shaped utilisation curve here is a schematic drawn to match the qualitative shape reported in the position-effect literature. The curve's exact height, depth, and asymmetry vary widely by model, task, and context length; treat it as a sign structure, not as reproduced measurements.

This is the sentence I would put above the desk of anyone building agents. The context window is a capacity guarantee and nothing more. Whether the model uses what is in there depends on how much else is in there and where it sits.

Memory is a loop, not a database

Most introductions to agent memory start with a vector database and a similarity search, which is a bit like explaining a kitchen by describing the fridge. The fridge matters. It is not the kitchen.

The useful abstraction is a closed loop, and it has three fundamental operations rather than one:

Write decides whether an observation deserves to become a memory at all, and in what form. Manage maintains the store over time: merging, superseding, consolidating, expiring, deleting. Read decides what comes back and how it is arranged for the next decision. A vector database implements a small part of Read and almost none of Write or Manage. That is why teams who buy a vector database and declare memory solved end up, six weeks later, with an agent that confidently tells users things they stopped believing in March.

The loop closes because the agent's own actions produce the next observations. This is what separates agent memory from document retrieval: the store is downstream of the agent's own behaviour. Bad decisions produce bad experiences, which get written, which get retrieved, which produce more bad decisions. We will come back to this in §16, where it has real teeth.

Notice how many decision points there are. Between the observation arriving and the model acting, an engineered system makes at least six choices: whether to remember, in what form, where to file it, what to merge it with, what to retrieve later, and how to lay it out. Each of those is a policy. Each can be wrong independently. A vector store gives you one of them.

Four kinds of memory

The standard taxonomy borrows from cognitive psychology. I want to be careful about the borrowing: the claim is not that agents reproduce human memory, and it is not that these four compartments are the uniquely correct decomposition. The claim is weaker and more practical — these categories have different lifetimes, different write triggers, different retrieval patterns, and different failure modes, so keeping them separate is good engineering hygiene. The vocabulary comes from Tulving's episodic/semantic distinction; the mapping onto language agents was formalised in the Cognitive Architectures for Language Agents framework, which sorts existing systems by which memory type they actually implement.

Working memory — what matters right now

The current goal, the plan, the file being edited, the last tool result, the unresolved question. This lives directly in context and is discarded or checkpointed at task boundaries. It is the closest thing an agent has to a program's stack frame.

goal      deploy the staging service
step      3 of 6 — fix failing test
editing   services/auth/token.py
last      pytest -k oauth  →  1 failed
error     AssertionError: scope 'read:user' missing
open      does the staging client have the scope?

The important design property here is that working memory should be rewritten, not appended. A plan that grows by accretion becomes a log; a plan that is edited in place stays a plan. This is the difference between an agent that knows what it is doing at step 40 and one that is reconstructing its intent from a transcript.

Episodic memory — what happened

Events and experiences, with time attached. Formally an episode is a tuple

\[ e_i \;=\; \big(s_i,\; a_i,\; o_i,\; r_i,\; t_i\big) \]

— state, action, observation, outcome, timestamp. For agents this is unusually powerful because entire trajectories can become memories, not just facts. “I tried X, it failed with Y, then Z worked” is a single episode carrying a full causal chain.

Generative Agents is the canonical demonstration: it stores a complete natural-language record of experiences and retrieves them for planning using a combination of relevance, recency, and importance. Reflexion is the other end of the same idea — it keeps reflective text in an episodic buffer specifically so that a later attempt at the same task can exploit what went wrong in the earlier one, with no weight update at any point.

Semantic memory — what I know

Distilled facts, detached from the episode that produced them. “The user works on vision-language models.” “Project X uses PostgreSQL.” “Service Y requires OAuth scope Z.”

The move from episodic to semantic is consolidation, and it is where most of the compression happens:

Note what was lost: the phrasing, the date, the fact that this came up while discussing something else. Note what was gained: retrievability by topic, updatability in place, and a 45× reduction. Consolidation is lossy compression with an editorial policy, and the policy is the interesting part.

Procedural memory — how to do something

This is the category that makes agents qualitatively different from chat assistants, and it is under-discussed. Procedural memory stores reusable capability: how to deploy this repository, how to recover this database, the shape of the query that works against this API, the fix for a build failure that recurs every few weeks.

Voyager is the clearest illustration. Its Minecraft agent turns successful behaviours into executable code stored in an ever-growing skill library, retrieved for later tasks and composed into more complex ones — and, critically, it interacts with the underlying model purely through prompting, with no parameter updates. The skills it accumulates are temporally extended, interpretable, and compositional, so capability compounds rather than resetting each episode.

That is the striking claim hiding in this section:

Reflective memory — what I have concluded

Arguably a fifth type, arguably a process that produces the other types. Raw experience says tool X failed on this file. Reflection says tool X fails on files above about 50 MB; use tool Y for those. The second is a generalisation, and it is far more valuable per token than the episode that produced it.

Generative Agents periodically synthesises higher-level reflections from accumulated experience and stores those alongside the raw stream, so retrieval can surface either. Reflexion turns a failure signal into text explicitly intended to change the next attempt. A-MEM pushes the idea further still: new memories are dynamically linked to related historical ones, and existing memory representations are allowed to evolve as new information arrives, so the store behaves less like an append-only log and more like a set of notes that get rewritten as understanding improves.

\[ \{e_1, \ldots, e_k\} \;\xrightarrow{\ \text{reflect}\ }\; m_{\text{semantic}} \;\text{or}\; m_{\text{procedural}} \]

How a memory is written down

Having decided that something should be remembered, you must decide as what. This choice determines everything downstream: what can be retrieved, what can be updated, what can be contradicted, and how much it costs to bring back.

There is a spectrum, and it trades fidelity against retrievability. Nothing on it dominates.

Full transcript

Lossless and useless. Everything is there; nothing is findable; the token cost is the entire history. Still worth keeping as an archival tier (see §13) because it is the only representation from which you can re-derive the others when your extraction policy improves.

Chunks

Classic retrieval segmentation: split by length or by structure, embed, index. Cheap and general. Its weakness for agents is that chunk boundaries are insensitive to event boundaries. A tool call, its failure, and its fix are one causal unit; a 512-token window will happily cut through the middle of it, leaving you with a chunk that says a thing failed and a separate chunk that says something was fixed.

Summaries

Compress 10,000 tokens into 500. Effective, and the workhorse of every production agent's compaction step. But summarisation is lossy in a specific and awkward way: it is lossy with respect to a purpose, and the purpose is not known at write time. Which leads to the question that this entire article circles:

Atomic facts

Triples of the form \((\text{subject},\ \text{relation},\ \text{object})\): (user, prefers, vegetarian food). Small, individually retrievable, and — crucially — individually updatable. You can supersede one triple without touching its neighbours. This is why systems that need to track changing state converge on fact-level granularity rather than passage-level.

Structured records

An atomic fact with its metadata attached:

{
  entity:     "project-x",
  fact:       "deploys via GitHub Actions, not Jenkins",
  type:       "semantic",
  source:     "user message 2026-08-14T11:02Z",
  confidence: 0.94,
  valid_from: "2026-08-14",
  valid_to:   null,
  scope:      "project",
  last_read:  "2026-09-01"
}

This is much closer to what production memory looks like, and every field earns its place. We will use valid_from/valid_to in §10, confidence and source in §17, scope for isolation, and last_read for forgetting policy.

Vector memory

Embed each memory, \(m_i \mapsto e_i \in \mathbb{R}^d\), retrieve by cosine similarity. Excellent for semantic association: it will find the memory about “authentication scopes” when you ask about “permission errors,” which no keyword index would. Weak at exactly three things, all of which matter for agents: exact temporal ordering, relational updates, and contradiction. An embedding of “the user lives in Hyderabad” and an embedding of “the user lives in Bangalore” sit right next to each other in the space. Similarity cannot tell you which one is current; it can only tell you they are about the same thing.

Graph memory

Entities as nodes, relations as edges, with time on the edges:

Zep is the clearest worked example of this design. Its core is Graphiti, a temporally-aware knowledge-graph engine that integrates both unstructured conversational data and structured business data while preserving historical relationships — the graph is organised into an episode subgraph holding raw non-lossy input, a semantic entity subgraph extracted from it, and a community subgraph above that. Mem0 explores both a straightforward extracted-memory pipeline and a graph variant for relational structure.

The honest assessment: graphs buy you multi-hop relational queries and clean temporal invalidation, and they cost you an extraction step that can itself be wrong, plus meaningfully more write-time latency and complexity. The published comparisons between graph and vector memory are mostly run by the authors of the graph systems on benchmarks of their own selection, which is not a criticism of anyone's integrity but is a reason to run your own evaluation before committing.

In practice, serious systems use several of these at once: a transcript archive, a fact table with provenance, a vector index over both, and increasingly a graph for entity relations. The choice is not which representation but which combination, and how they stay consistent with each other.

The write problem

Now the central claim of the article. Almost all popular material on this topic is about retrieval. Retrieval is the easier half.

When you retrieve, you have the question in hand. You can embed it, expand it, filter on it, rerank against it. The objective is well posed. When you write, you must decide whether a piece of information will be useful for a query that has not been asked yet, drawn from a distribution you do not know. That is a fundamentally harder problem, and it is not hard because of engineering — it is hard because of information.

Write it out. The value of storing memory \(m\) is its expected marginal contribution across the unknown future query distribution, minus what it costs to carry:

\[ V(m) \;=\; \mathbb{E}_{q \sim \mathcal{Q}}\Big[\, P\big(m \in R(M, q)\big)\cdot \Delta\big(m \mid C_q\big) \,\Big] \;-\; \lambda\,\ell(m) \;-\; \gamma\,D(m) \]

where \(D(m)\) is the dilution cost — the damage \(m\) does when it is retrieved for a query it should not have matched. Store when \(V(m) > 0\).

Every term here is a problem. \(\mathcal{Q}\) is unknown and non-stationary: users change what they ask about. \(P(m \in R)\) depends on the retriever you have not finished building. \(\Delta\) depends on what else will be in context at that future moment. And \(D(m)\) is the term everyone forgets, which is why naive systems accumulate memories that are individually harmless and collectively toxic.

So write policies are heuristic by necessity. The useful move is to decompose the heuristic into signals you can actually estimate at write time:

The task utility row is the one that separates good systems from bad ones, and it is the one with the strongest empirical support. Xiong and colleagues studied exactly this — the effect of memory addition and deletion policies on long-term agent performance — and advocated a framework combining selective addition with a combined deletion policy, reporting roughly a 10% absolute average gain over baseline policies across three agents on three tasks, and validating it further under task distribution shift and constrained memory size. The mechanism is simple: gate writes on whether the episode actually succeeded, judged by an evaluator, rather than storing everything the agent did.

A worked triage

Consider an ordinary five minutes of conversation and what a sane policy does with each line.

Two rows in that figure deserve comment. The joke — “my favourite pizza today is pepperoni… actually I hate pepperoni, I was kidding” — is the canonical example of why a write policy cannot be a single-message classifier. The correct behaviour requires reading the retraction, which arrives after the write decision would naively have been made. Real systems handle this by deferring extraction to session end, or by allowing later messages to invalidate earlier writes, which is the same mechanism as the update problem in the next section.

And the credential: never persist is not a heuristic, it is a hard rule. A memory store is a durable, replicated, retrievable, model-readable artifact. Anything that goes into it should be assumed to come back out in a context window somewhere, at some point, in front of someone.

Update is not append

Here is the smallest possible bug that breaks a memory system. Suppose the store contains:

m₁  user lives in Hyderabad

and later the user says they have moved. An append-only system now contains:

m₁  user lives in Hyderabad
m₂  user lives in Bangalore

Both are semantically similar to the query “where does the user live?” — in fact they are nearly identical in embedding space, since they differ by one token in the same syntactic slot. Retrieval returns both. The model must now guess. It will often guess by recency-in-context or by ordering, neither of which is a fact about the world.

The fix is to stop storing facts and start storing assertions with validity intervals:

\[ m \;=\; \big(\text{fact},\; t_{\text{start}},\; t_{\text{end}}\big) \]

m₁  lives_in(user, Hyderabad)   valid [2023-06-01 → 2026-08-31]
m₂  lives_in(user, Bangalore)   valid [2026-09-01 → ∞      ]

Now “where does the user live” is answerable (filter on \(t_{\text{end}} = \infty\)) and “where did they live last year” is also answerable, which the naive delete-and-replace design would have destroyed. This is the same insight as bitemporal modelling in databases, and it is worth being explicit that there are genuinely two clocks: when the fact was true in the world, and when the system learned it. They come apart constantly — a user tells you in September about a move that happened in July — and conflating them produces a memory that cannot answer “what did we believe at the time we made that decision?”

So the memory store needs a proper operation set, not just insert:

The distinction between SUPERSEDE and INVALIDATE matters more than it looks. “They moved” and “we were wrong” have different consequences for everything downstream that was decided using the old value. One is a change in the world; the other is a defect in the memory, and it should probably trigger a review of whatever else came from the same source.

LongMemEval is the benchmark that made this concrete for the field. It evaluates five long-term memory abilities of chat assistants — information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention — across 500 curated questions embedded in scalable chat histories, and reported commercial assistants and long-context models dropping around 30% in accuracy on retaining information across sustained interaction. Note that knowledge updates and temporal reasoning are two of the five: the field's own benchmark says update is not a footnote.

Forgetting is a feature

The default design has no deletion. Memories accumulate: \(|M_t| \to \infty\). This feels safe. It is not.

Let me give you the cleanest argument I know, because it is short and it is quantitative.

The 1/λt law

Model an append-only store. New facts arrive at rate \(a\) per unit time. Each fact stays true for an exponentially distributed duration with rate \(\lambda\) — that is, facts go stale at a constant hazard. At time \(t\), the store holds

\[ N(t) \;=\; a\,t \qquad \text{records} \]

of which the number still true is

\[ N_{\text{true}}(t) \;=\; a \int_0^t e^{-\lambda s}\,\mathrm{d}s \;=\; \frac{a}{\lambda}\Big(1 - e^{-\lambda t}\Big) \;\xrightarrow[t \to \infty]{}\; \frac{a}{\lambda}. \]

The number of true facts saturates. The number of records does not. So the fraction of your memory that is currently true is

\[ \boxed{\;\rho(t) \;=\; \frac{N_{\text{true}}(t)}{N(t)} \;=\; \frac{1 - e^{-\lambda t}}{\lambda t} \;\approx\; \frac{1}{\lambda t}\quad (\lambda t \gg 1)\;} \]

An append-only memory's truth density decays as \(1/t\). Not asymptotically-fine, not eventually-stable — it goes to zero. After ten fact-lifetimes, roughly one record in ten is current. After a hundred, one in a hundred. Retrieval will keep returning \(k\) items and most of them will be archaeology.

The assumptions are strong and worth naming: a constant arrival rate, a single staleness hazard for all facts, and no correlation between staleness and retrievability. Real memories have wildly heterogeneous half-lives — a person's name is effectively permanent, their current sprint is stale in two weeks — which is precisely the argument for per-fact stability estimates and expiry, rather than one global policy. The heterogeneity changes the constant; it does not change the direction.

Curves are computed live from the closed-form model above with the slider values; they are a model of the mechanism, not measurements from a deployed system.

The second argument: precision at fixed k

Even if nothing went stale, growth alone hurts. Retrieval returns a fixed \(k\). Suppose relevant memories score \(\mathcal{N}(\mu_r, \sigma^2)\) and irrelevant ones \(\mathcal{N}(\mu_i, \sigma^2)\) with \(\mu_r > \mu_i\). The expected number of irrelevant memories scoring above a given threshold is proportional to the store size \(n\). Hold \(k\) fixed and grow \(n\): the number of distractors competing for those \(k\) slots grows linearly, while the number of genuinely relevant items does not. Precision@k falls even with a perfect, unchanging world. Scale is itself a source of noise.

So: a deletion policy

\[ P\big(\text{delete} \mid m,\, t,\, s\big) \]

driven by age, access frequency, supersession, explicit expiry, confidence decay, redundancy against newer records, and — non-negotiably — user deletion requests. The Xiong et al. study is again the empirical anchor here: they combine selective addition with a deletion policy specifically to mitigate error propagation and misaligned experience replay, and validate the necessity under memory-size constraints. Which gives the section its slogan:

An agent with no memory of a past incident asks. An agent with a wrong memory of it acts.

Which memory comes back

Now the half everyone starts with. The simplest retrieval score is cosine similarity in embedding space:

\[ s_{\text{sem}}(m, q) \;=\; \cos\!\big(e_q,\, e_m\big) \]

and it is a genuinely good signal — but it is exactly one signal, and it answers exactly one question: is this memory about the same topic as the query? Several other questions matter as much.

Generative Agents' retrieval function is the canonical multi-signal design: it combines relevance (embedding similarity), recency (exponential decay since last access), and importance (a model-assigned score for how significant the memory is), and ranks by their weighted sum. Generalising:

\[ S(m \mid q,\, t) \;=\; \alpha\,R_{\text{sem}} \;+\; \beta\,R_{\text{rec}} \;+\; \gamma\,R_{\text{imp}} \;+\; \delta\,R_{\text{task}} \;+\; \eta\,R_{\text{temp}} \]

Why a weighted sum, of all things?

It looks like an arbitrary engineering hack, and it is usually presented as one. It is not. Suppose you want \(P(\text{useful} \mid \text{features})\) and you are willing to treat the features as conditionally independent given usefulness. Then by Bayes,

\[ \log \frac{P(\text{useful} \mid f_1..f_k)}{P(\lnot\text{useful} \mid f_1..f_k)} \;=\; \log\frac{P(\text{useful})}{P(\lnot\text{useful})} \;+\; \sum_j \log \frac{P(f_j \mid \text{useful})}{P(f_j \mid \lnot\text{useful})} \]

— a constant plus a sum of per-feature log-likelihood ratios. The weighted sum is the naive-Bayes posterior log-odds, and the weights \(\alpha, \beta, \gamma, \ldots\) are calibration constants for each feature's discriminative power. That reframing is useful in three ways: it tells you the features should be roughly independent (recency and last-access-time are not, so do not include both), it tells you the right way to fit the weights is against a labelled set of “was this memory actually useful,” and it tells you the score is a log-odds, so it is meaningful to threshold it rather than always taking top-\(k\).

That last point is underrated. Always returning \(k\) memories means that when nothing relevant exists, you return \(k\) irrelevant ones. A calibrated threshold lets the system return nothing — which is the retrieval-side prerequisite for abstention, one of LongMemEval's five abilities.

Feature values for the eight memories are hand-authored to illustrate how the weights trade off; the ranking is computed live from your slider settings.

The query is itself a decision

Here is a failure mode that no amount of retriever tuning fixes. The user says:

"can we do it like last time?"

Embed that literally and search. You will retrieve memories about the phrase last time. You will retrieve nothing useful, because the semantic content of the query is almost entirely absent from its surface form — it lives in the conversation state.

What the agent actually needs to ask memory is something like:

resolved memory query
task      deployment of service `auth-gateway`
need      the procedure used for the last successful
          deploy of this repo to the staging environment
filters   entity=auth-gateway, env=staging,
          outcome=success, type=procedural
order by  recency

So retrieval takes a preceding step:

\[ q_{\text{mem}} \;=\; f\big(q_{\text{user}},\; s_t,\; \text{goal}\big) \]

and \(f\) is not a string transformation — it is a decision the agent makes, often with a model call, using working memory to resolve references, infer the implicit entity, and choose filters. Framed properly, querying memory is an action in the agent's action space, on the same footing as calling a tool. Which means it can be planned, it can be retried with a different formulation, and it can be evaluated on its own.

This is also where hybrid retrieval earns its keep. The resolved query above has a semantic part (“deployment procedure”), an exact-match part (repository name), a metadata filter (environment, outcome), a temporal constraint (most recent), and potentially a graph traversal (what does this service depend on). No single index does all five. Production memory retrieval is a small query planner, not an embedding lookup.

The memory hierarchy

Computer architecture solved a structurally identical problem fifty years ago, and the analogy is not decorative — it is the actual design.

A CPU has a small, fast, expensive store and a large, slow, cheap one, and the operating system creates the illusion of a large fast store by moving data between them on demand. An agent has a small, expensive context window and a large, cheap external store, and the memory system creates the illusion of a large context by moving information between them on demand.

MemGPT makes this explicit. It proposes virtual context management, drawing directly on hierarchical memory in operating systems, and gives the model functions to page information between a limited main context and larger external context — including a system-generated memory-pressure alert that prompts the model to write things out before the window fills. The model manages its own memory through tool calls, and control flow between model and system is mediated by interrupts. It is an operating system for a context window, and the framing has held up.

The tiers, and what actually lives in each:

Two design consequences follow from taking the analogy seriously. First, keep the archive. It is cheap, and it is the only tier from which you can re-derive the others when your extraction policy improves — the equivalent of keeping raw data so you can re-run the pipeline. Zep's design has exactly this property: the episode subgraph is a non-lossy store from which semantic entities and relations are extracted, so the extraction can be redone. Second, the eviction policy is where the intelligence is, exactly as it is in a cache. Choosing what to page out is choosing what the agent will find hard to think about next.

Constructing the context

Retrieval hands you thirty candidate memories. The budget fits eight. Selection was §4. What remains is layout — and because utilisation depends on position, layout is not cosmetic.

Ordering

A defensible default, and the reasoning behind each choice:

1  system + policy          stable → prefix-cacheable, and instructions
                            at the head are the best-attended region
2  tool definitions        stable within a session
3  durable user/project    changes rarely; safe to cache
   facts
4  current goal + plan     the thing everything else serves
5  retrieved memories      clearly delimited, marked as memory
6  recent conversation     volatile
7  latest tool output      most volatile
8  current user message    the other best-attended region

Two forces are being balanced here and it is worth separating them, because they occasionally conflict.

The caching force says: order by volatility, most stable first. Any token before the first change is a prefix-cache hit; any token after it must be recomputed. Putting a volatile timestamp in the system prompt invalidates the entire cache on every call, which is a surprisingly common and expensive bug.

The attention force says: put what matters at the ends. Instructions at the head, the immediate request at the tail, and everything that is merely context in between. This is a direct engineering consequence of the position-effect literature: if the middle is where utilisation dips, do not put load-bearing information there.

They agree more often than they conflict, because stable things (policy) are usually also the things you want at the head. Where they conflict — a critical retrieved memory that would sit in the middle — the standard remedy is to restate: put the memory in its natural slot and repeat the one line that matters near the query. Restating a 15-token fact to buy better utilisation is an excellent trade.

Marking provenance in the prompt

Retrieved memories should never be spliced into the conversation as though the model said them. They should be delimited, attributed, and dated:

<memories retrieved="3" as_of="2026-09-05">
  [user, stated 2026-08-14, confidence high]
    prefers long-form technical writing
  [tool-verified 2026-09-01, confidence high]
    project-x deploys via GitHub Actions
  [inferred 2026-08-30, confidence LOW — verify before acting]
    the staging cluster may still be on the old runner image
</memories>

This costs a handful of tokens per memory and buys three things. The model can weight a low-confidence inference differently from a user's direct statement. It can cite where a claim came from when the user asks. And it can notice when a memory is stale relative to the current date, which it cannot do if you strip the timestamps.

Compression, and its bill

When the candidate context exceeds the budget, \(L(C) > B\), there are four moves, in increasing order of information loss:

1. Drop. Remove low-marginal-value items. Zero distortion on what remains — you simply do not have it. Always try this first; it is the only lossless option in the sense that nothing surviving is corrupted.
2. Extract. Replace a document with the facts from it that bear on the current task. Very high compression, distortion concentrated in what you failed to anticipate.
3. Summarise. Rewrite a span more briefly. This is the compaction step every long-running agent needs, and it is where subtle corruption enters: a summary that says “the tests passed” when three passed and one was skipped is not detectably wrong from inside the context.
4. Token-level compression. LLMLingua's approach: use a small language model to score tokens and drop the ones that carry least information, with a budget controller allocating different compression ratios to different parts of the prompt and an iterative token-level pass that accounts for interdependence between what is kept. It reports up to 20× compression with limited loss on its evaluation suite; LongLLMLingua extends it to long-context settings and targets position bias explicitly. The compressed prompt is often unreadable to humans while remaining usable by the model, which is a strange and instructive fact about where redundancy sits in natural language.

The thing to internalise is that all four are codecs with unmeasured distortion. You would never ship an image pipeline without looking at the output. Compression steps in agents routinely ship without any measurement of what they destroyed. If your agent has a compaction step, it needs a test: run a set of questions answerable from the pre-compaction context and check they are still answerable after. That single test catches an enormous class of long-horizon failures where the agent “forgets” a constraint it was told 200 turns ago — not because the context overflowed, but because a summariser decided the constraint was not important.

Memory, RAG, and weights

Three ways to make knowledge available to a model, constantly confused with each other because two of them share infrastructure.

Memory is not RAG

They can run on the same vector database. They solve different problems.

The one-line version:

The last row of the table is the one with real consequences. RAG's corpus is exogenous: it does not degrade because the retriever is bad. Memory's corpus is endogenous. A bad agent writes a bad store which makes it worse. That closed loop is the subject of the next section.

Memory is not fine-tuning

Suppose the agent discovers that a particular API requires a region parameter. Two ways to make that stick.

That last-but-one row is the honest limitation of the memory approach and it does not get said enough. A weight update generalises: the model may apply the pattern to a case that shares no surface features with the one it learned from. A memory write does not generalise at all — it applies exactly where the retriever fires and nowhere else. Reflection is the partial remedy, since a reflection is a deliberately generalised memory, but a written generalisation only covers the cases the writer thought of.

Reflexion is the cleanest existence proof that the memory route works at all: it reinforces language agents not by updating weights but through linguistic feedback, keeping reflective text in an episodic buffer to improve decisions on subsequent trials, and reports substantial improvements across sequential decision-making, coding, and reasoning tasks. Voyager's growing skill library is the procedural counterpart. Both change behaviour durably. Neither touches \(\theta\).

They are complements, not competitors. Weight updates are how you install a capability; memory is how you install a fact, a preference, or a specific procedure. Using either for the other's job is the mistake.

When memory poisons itself

The closed loop is the reason agent memory has failure modes that document retrieval simply does not have. The agent writes what it later reads. If the write policy is not discriminating, errors do not average out — they compound.

The empirical anchor is Xiong and colleagues' study of memory management. They report that LLM agents exhibit an experience-following property: when a task input is highly similar to the input stored in a retrieved memory record, the agent's output tends to be highly similar to that record's output. This is exactly the behaviour you want when the record is good. It is exactly the behaviour that destroys you when it is not, and the paper names the two resulting challenges directly — error propagation, where inaccuracies in past experiences compound and degrade future performance, and misaligned experience replay, where seemingly correct executions provide limited or misleading value as experiences.

A toy model of the compounding

Let me make the mechanism quantitative, with the caveat stated up front: this is a model of the mechanism, chosen for transparency, not a fit to anyone's measurements.

Let \(\varepsilon_0\) be the agent's error rate with no memory. Let \(\phi \in [0,1]\) be the experience-following coefficient — the probability that the agent's output mirrors a retrieved record rather than being reasoned afresh. Let the write gate accept a correct episode with probability \(\beta\) and an incorrect one with probability \(\alpha\), and define the gate's leakage ratio

\[ \rho \;=\; \frac{\alpha}{\beta}. \]

\(\rho = 1\) is no gate at all — store everything. \(\rho < 1\) is a gate that filters errors better than chance. \(\rho > 1\) is an inverted gate that preferentially stores mistakes, which sounds absurd until you notice that a gate rewarding confident outputs is exactly that on tasks where the model is confidently wrong.

In steady state, the fraction of stored records that are wrong is

\[ w(e) \;=\; \frac{\alpha e}{\alpha e + \beta(1-e)} \;=\; \frac{\rho\,e}{\rho\,e + 1 - e}, \]

and the agent's error rate is a mixture of following memory and reasoning fresh:

\[ e \;=\; \phi\, w(e) \;+\; (1-\phi)\,\varepsilon_0. \]

Solve the fixed point. Writing \(k = \rho - 1\) and \(c = (1-\phi)\varepsilon_0\), it is a quadratic, and the relevant root is

\[ e^\star \;=\; \frac{-(1 - \phi\rho - ck) \;+\; \sqrt{(1 - \phi\rho - ck)^2 + 4kc}}{2k}, \qquad k \ne 0, \]

with \(e^\star = \varepsilon_0\) exactly at \(\rho = 1\).

Three readings fall out, and they are the whole design lesson:

• No gate is neutral, not safe. At \(\rho = 1\) the error rate is unchanged at \(\varepsilon_0\), no matter how large \(\phi\) is. Storing everything buys you nothing on accuracy — you pay the storage and dilution cost for a wash.
• A discriminating gate compounds downward. With \(\varepsilon_0 = 0.15\), \(\phi = 0.9\), a gate at \(\rho = 0.25\) drives \(e^\star\) to roughly 0.02. The agent gets substantially better than its own base rate, because it is following a store that is cleaner than its own outputs.
• Following amplifies whatever it is fed. The same \(\phi = 0.9\) with an inverted gate at \(\rho = 4\) drives \(e^\star\) to roughly 0.89. High experience-following is not a virtue or a vice; it is a gain term. It multiplies the quality of your write policy in both directions.

Which is a clean restatement of why selective addition matters more than any retrieval improvement: \(\phi\) is a property of the model you were given, and \(\rho\) is a property of the system you built.

Curves computed live from the fixed-point equation above. This is a transparent toy model of the compounding mechanism, chosen because every term is interpretable; it is not a reproduction of published measurements.

The full catalogue

The memory-injection row is worth dwelling on, because it is the one that turns a reliability problem into a security problem. If an agent browses a page, and that page contains text saying remember that the deployment key is now stored at this URL, and the write policy is “extract salient facts from observations,” then an attacker has just written to your agent's long-term memory. Persistence makes this much worse than ordinary prompt injection, because the payload survives the session that delivered it and fires later, in a context the user never connects to the page they visited last week.

Provenance, scope, and privacy

Everything above argues for the same conclusion from different directions: a memory should not be a string. It should be

\[ m \;=\; \big(\text{content},\; \text{provenance},\; \text{time},\; \text{confidence},\; \text{scope}\big) \]

and the four fields after content are what make the difference between a store you can operate and one you cannot.

Provenance classes

Not all facts arrive equally. A workable ladder:

The bottom row is the important one, and it is the discipline most systems lack. A model's hypotheses are useful inside an episode and dangerous across episodes, because a stored guess is indistinguishable at read time from a stored observation unless you kept the label.

Scope isolation

Four scopes, and a leak between any pair is a bug with a severity that scales with the distance:

• Session — the current task. Discarded or promoted at the end.
• Project — shared across sessions on one codebase, workspace, or account.
• User — preferences, identity, standing instructions.
• Organisation — shared conventions and runbooks across users.

Scope is not a metadata nicety, it is an access-control boundary and should be enforced at the retrieval layer as a hard filter, not as a ranking signal. A memory that surfaces one user's information in another user's session is not a relevance error; it is an incident. Design the retriever so that scope is applied before scoring and cannot be outranked.

Deletion has to be real

If a user asks the agent to forget something, that must mean removed from the store, removed from the vector index, removed from the graph, removed from any derived summary that quoted it, and absent from future contexts. Derived artifacts are the part teams miss: deleting the source fact while a rolling summary still contains it means the agent keeps saying the thing it was told to forget, from a copy nobody indexed. Any consolidation step needs to record which source records it drew on, precisely so that deletion can chase the derivation.

How do you know it works?

The naive evaluation is “did it remember my name.” That measures one of the easiest of the abilities that matter, and passing it tells you almost nothing.

Two benchmarks have done the useful work of enumerating what should be measured. LongMemEval defines five core long-term memory abilities for chat assistants — information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention — and breaks the design space into indexing, retrieval, and reading stages. MemoryAgentBench comes at it from memory science and identifies four core competencies for memory agents: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Its methodological contribution is arguably as important as its taxonomy: it converts long-context datasets into genuinely incremental multi-turn form, because real agents accumulate information over turns rather than receiving it in one block, and it reports that current methods fall short of mastering all four competencies.

The two lists overlap and disagree in instructive ways. LongMemEval cares about abstention; MemoryAgentBench cares about selective forgetting. Both are about the system's willingness to not do something, which is the capability that naive evaluation never touches.

An evaluation taxonomy you can build against

The evaluation that actually matters: causality

Suppose your agent-with-memory scores 90% and your agent-without scores 78%. Did memory cause the difference? Not necessarily — the memory-enabled configuration usually differs in more than one way, and the model may have known some answers regardless.

The fix is an ablation ladder. Run the same task set under five conditions and read the gaps between them, because each gap has a distinct interpretation:

1. No memory. The floor. Whatever the model can do from parametric knowledge plus the current turn.
2. Random memories. Same token budget, same format, irrelevant content. The gap from (1) to (2) measures pure distraction damage, and it is often negative — which is the single most useful number in the ladder, because it prices the cost of a bad retrieval.
3. Retrieved memories. Your actual system.
4. Oracle memories. The gold set, hand-labelled, in the same format and budget. The gap from (3) to (4) is your retrieval headroom: how much is left on the table by the retriever, as opposed to by the writer or the reader.
5. Full history. Everything in context, budget permitting. If (5) beats (4), your memory representation is destroying information that the raw history preserved — a write problem, not a read problem. If (4) beats (5), you have direct evidence that curation beats capacity.

Decompose the total as

\[ \underbrace{A_4 - A_1}_{\text{value of the right memories}} \;=\; \underbrace{(A_3 - A_1)}_{\text{delivered lift}} \;+\; \underbrace{(A_4 - A_3)}_{\text{retrieval headroom}} \]

and separately track \(A_2 - A_1\), the distraction cost, and \(A_5 - A_4\), the compression loss. Four numbers, four different teams' worth of work localised.

Bar heights are an illustrative pattern showing how the five conditions relate and what each gap localises. They are not measurements; the point is the shape of the decomposition, and your own numbers will differ.

One more control worth running, because it is cheap and it catches a whole class of self-deception: did the model actually use the retrieved memory, or did it already know? Take the questions your system answers correctly with memory, and re-ask them with memory removed but everything else identical. The ones it still gets right were never memory wins.

The whole machine

Assembling everything into one picture. Nothing here is new at this point — every box has been argued for — but seeing the data flow end to end makes the dependencies concrete, and in particular makes clear how few of the boxes a vector database occupies.

If you were building this from nothing, a defensible order of construction — each step earning its complexity from a failure you have actually observed — would be:

1. Keep the raw transcript archive. Cheap, and everything else can be re-derived from it.
2. Add session-level compaction with an explicit test that post-compaction context still answers pre-compaction questions.
3. Add fact extraction with provenance, timestamps, and scope. Store structured records, not strings.
4. Add supersession. The moment you have two facts about one slot, you need validity intervals.
5. Add outcome-gated writes. This is the highest-leverage single change, per the memory-management literature.
6. Add multi-signal retrieval with a calibrated threshold, so the system can return nothing.
7. Add deletion and expiry, driven by supersession, age, and access.
8. Add consolidation and reflection — episodes into rules, rules into procedures.
9. Add a graph layer only when you have a concrete multi-hop relational query that your fact table cannot answer.

Most systems that struggle have done step 9 and skipped steps 3 through 7.

What is still open

It would be dishonest to end a first-principles article with a solved problem. Several of the load-bearing questions here are genuinely unsettled, and I would rather name them than paper over them.

The two sentences

If the article compresses to anything, it compresses to this.

The context window is a channel with a fixed width and a non-uniform response. The history is unbounded. Everything interesting is the codec in between — and the hardest part of the codec is not the decoder that fetches, but the encoder that decides, without knowing what will be asked, what is worth keeping.

References

Grouped by topic. Every entry links to its primary source.