Skip to content

Persistent Memory for an LLM App

This pattern addresses limitations #1 and #2 together: a model has no memory beyond its context window (it forgets everything once a session ends), and it doesn't learn between runs. If your product needs to accumulate knowledge about a project, a user, or a long-running task — and reuse it next time — you have to build that memory. The model won't keep it for you.

For non-engineers: the model is like a brilliant consultant with total amnesia — sharp in the moment, but every conversation starts from zero. Persistent memory is the file cabinet you keep for them, so each new engagement starts with everything learned so far already on the desk.

What "memory" means for a product

Don't confuse two different things:

  • Conversation context — what's in the current window. Ephemeral; gone when the session ends; bounded by the window size.
  • Persistent memory — durable state your application stores (in a database, on disk, in object storage) and deliberately re-injects into future context windows.

Persistent memory is not a model feature. It's an application feature: you decide what's worth remembering, store it, and feed the relevant slice back in when needed.

Case study: AuditAgent's project memory

When AuditAgent scans the same repository repeatedly, it shouldn't re-derive everything each time. It maintains a per-project knowledge base that captures, curates, and reuses context across scans. The design is a clean template for product memory in general:

Two memory zones, deliberately separated:

  • Long-term memory (archive). Durable knowledge about the project — security invariants, threat-model hypotheses, contradictions, an overview, user-provided instructions, findings the user has explicitly excluded. Stored in a database, keyed per project, and shared across everyone on the same team (it's org-scoped, so a teammate's curated knowledge benefits the whole org).
  • Short-term / ephemeral memory. Per-run scratch state — candidate findings, rejected findings, working notes. Checkpointed during the run (so a crashed job can resume) but dropped afterward and never promoted to the archive.

This split is the key design decision. Mixing "durable, curated truth" with "this run's scratch work" is a recipe for polluting your knowledge base with noise.

How memory reaches the model

A subtle but important choice: AuditAgent doesn't inject memory by string-concatenating it into a prompt. Instead, at the start of each run it materializes memory to markdown files inside the working directory — and writes a root context file that points at them. The agents then read those files (and can re-read them mid-task with a Read tool).

Why files instead of prompt injection?

  • Inspectable after the fact — you can open the files and see exactly what context the model had during a run. Invaluable for debugging.
  • Re-readable on demand — a tool loop lets the model pull a specific memory file only when it needs it, instead of front-loading everything into the window.
  • Same token cost, more control — and it composes with retrieval: memory files get indexed alongside code, so the right invariant surfaces next to the relevant function.

Notice the symmetry with the assisted-engineering side: a CLAUDE.md/AGENTS.md pointing at focused docs is the exact same idea — durable, curated, on-disk context the agent loads only as needed. The pattern is universal; only the consumer (a product's agents vs. your IDE assistant) differs.

Curating memory: append, dedupe, and review

Memory that only grows becomes noise. AuditAgent curates it:

  • Merge, don't just append. New knowledge from a run is merged with the archive, with semantic deduplication so the same invariant isn't stored ten times.
  • Source priority. User-provided knowledge outranks machine-generated knowledge on conflict — humans are authoritative.
  • Human review gate. Machine-proposed additions to long-term memory are staged as pending changes for a person to accept or reject, rather than silently mutating the archive. (This mirrors the advice to review AI-generated memory before committing it.)
  • Append-only on disk, removal only by explicit action. Stale entries disappear only when a human removes the underlying item — never by a destructive "rebuild" that could wipe prior context.

The API's first-party memory tool

Anthropic's Claude API ships a first-party memory tool built for exactly this cross-session problem. You declare it ({"type": "memory_20250818", "name": "memory"}) and the model manages a client-side memory directory itself through a fixed command set — view, create, str_replace, insert, delete, rename. Crucially, you still own the storage backend: the tool standardizes the read/write protocol, but where those files live (disk, database, object store) and any access control are your code — the same "memory is an application feature, not a model feature" point from the top of this page, just with the interface handed to you. It composes cleanly with the file-materialization pattern above.

For context that grows within a single long-running session (rather than across sessions), the API offers two related-but-distinct tools: context editing, which clears stale tool results and thinking blocks, and compaction, which summarizes earlier history server-side. Memory is for persistence across runs; those two are for staying under the window during one run. See Anthropic's memory tool docs.

A checklist for product memory

If you're adding memory to an LLM feature, decide each of these explicitly:

  • What's worth remembering? Curated facts, not raw transcripts.
  • Two zones? Separate durable/curated memory from per-run scratch.
  • Scope/keying. Per user? Per project? Per team/org? (Team-shared memory multiplies value but needs access control.)
  • How does it re-enter context? Files + retrieval, or direct injection? Will it fit the budget?
  • Dedup & conflict rules. How do you keep it from growing into noise? Who wins on conflict?
  • Human review. Does a person approve what gets written to long-term memory?
  • Failure mode. If memory is unavailable, does the feature degrade gracefully or break? (AuditAgent continues with reduced context rather than failing the run.)

Takeaways

  • The model has no memory across sessions and doesn't learn — persistent memory is your job, not the model's.
  • Separate durable, curated memory from ephemeral per-run state.
  • Materializing memory to files (vs. prompt injection) makes it inspectable, re-readable, and retrievable — and it's the same pattern as CLAUDE.md/AGENTS.md on the tooling side.
  • Curate aggressively: merge, dedupe, prioritize human input, and gate long-term writes behind review.

Sources