Memory tiers, long-term user memory, context budgets, compaction, file-based scratchpads.
Conversation memory tiers
Recent turns verbatim, older turns as a rolling summary, everything recallable by vector search.
Sketch this when: Sketch when asked how a long conversation fits in a finite context window.
- Recency gets fidelity, history gets compression, everything stays findable.
- Summaries lose detail silently — the vector tier is the safety net.
- Tier budgets are tunable per product: support bots weight recall, chat weights recency.
Deep dive: Short-term memory →Mermaid source
graph TD
turns[Incoming turns] --> recent[Tier 1: last N turns verbatim]
turns --> summ[Tier 2: rolling summary of older turns]
turns --> vec[(Tier 3: vector index over full history)]
q[Current query] --> vec
vec --> recall[Relevant old snippets]
recent & summ & recall --> ctx[Assembled context]
ctx --> llm[LLM]
Long-term user memory
Extract durable facts after each conversation; retrieve them into the system prompt next session.
Sketch this when: Sketch when personalization across sessions comes up.
- Extract facts, not transcripts — 'prefers TypeScript' beats 500 stored turns.
- Contradiction handling is the hard part: new facts must supersede stale ones.
- Memory is user data — deletion and inspection are product requirements.
Deep dive: Agent memory →Mermaid source
graph LR
conv[Conversation ends] --> extract[LLM extracts durable facts]
extract --> dedupe{New, duplicate, or contradiction?}
dedupe -->|new| store[(Memory store)]
dedupe -->|update| store
next[Next session] --> retr[Retrieve relevant memories]
store --> retr
retr --> sys[Inject into system prompt]Context-window budget
The assembly diagram: every context slot has a token budget and an eviction order.
Sketch this when: Sketch when asked what happens as the prompt grows past the window.
- Context is a zero-sum budget; every slot fights every other slot.
- Decide eviction order up front — accidental truncation drops the wrong thing.
- More context isn't free even when it fits: attention degrades and cost scales.
Mermaid source
graph LR
subgraph window[Context window]
sys[System ~5%] --> tools[Tools ~10%]
tools --> mem[Memory ~10%]
mem --> docs[Retrieved docs ~30%]
docs --> hist[History ~40%]
hist --> q[Query ~5%]
end
evict[Eviction order: history first, system never] -.-> hist
q --> llm[LLM]Context compaction loop
Near the limit, summarize the transcript into goals + state and continue with a fresh context.
Sketch this when: Sketch when asked how an agent works for hours without running out of context.
- The summary must preserve intent and open threads, not just topics.
- Compaction loses detail — pair it with a file scratchpad for anything precise.
- This is how long-running coding agents actually survive.
Mermaid source
graph TD
run[Agent running] --> check{Context near limit?}
check -->|no| cont[Continue]
cont --> run
check -->|yes| compact[Summarize transcript keep goals, state, files]
compact --> fresh[Fresh context summary + recent turns]
fresh --> runFile-based scratchpad memory
The agent writes notes, todos, and artifacts to a workspace and re-reads them across steps.
Sketch this when: Sketch when asked where an agent keeps precise state the context window can't hold.
- Files survive compaction, crashes, and handoffs — context doesn't.
- The todo file doubles as the plan; the agent literally checks items off.
- This is the blackboard pattern applied to a single agent.
Deep dive: Agent memory →Mermaid source
graph TD
agent[Agent] -->|writes| notes[notes.md findings so far]
agent -->|writes| todo[todo.md remaining work]
agent -->|writes| art[Artifacts code, data]
notes & todo & art --> ws[(Workspace files)]
ws -->|re-reads next step| agent
compact[Context compaction] -.->|survives it| ws