Personal Memory System

Learning Objectives

By the end of this project, you will be able to:

• Explain why an LLM is stateless and why a memory system is the layer that makes an assistant remember across turns and sessions.
• Describe the memory stream: a growing store of memory objects, each with a creation time, a last-accessed time, and an importance score.
• Implement the retrieval score that combines relevance (semantic similarity), recency (exponential time decay), and importance (salience) into one number.
• Implement exponential recency decay, recency = decay_rate ^ (hours since last access), and explain why retrieval refreshes a memory's recency.
• Distinguish episodic (events, dated), semantic (durable facts), and procedural (skills) memory, and why tagging by kind beats one flat store.
• Explain the memory hierarchy — small in-prompt main context vs large out-of-prompt external context — and frame retrieval as paging into a limited budget.
• Name and reproduce the major failure modes of a memory system and say which signal caused each.

1. Motivation

Why This Exists

An LLM is stateless: each call starts cold, and the context window is finite. Project 01's chatbot only "remembered" because you replayed the entire history into every prompt — that works for one short conversation and then collapses: histories outgrow the context window, every replayed token costs money, and nothing survives across sessions. Close the app and the assistant forgets you. A memory system is the layer that fixes this: store experiences durably outside the prompt, and retrieve only the relevant few back in when they matter.

What Breaks Without It

The failure is quiet and corrosive: an assistant that forgets a standing instruction you gave it (no write-back), or fixates on an old irrelevant detail because it scored relevance only (no recency/importance), or contradicts a decision you made yesterday (never retrieved it). Users don't file a bug — they just stop trusting it.

Real-World Stakes

Every assistant that "remembers you" — ChatGPT's memory, coding copilots that recall your conventions, support bots that know your history — is some version of this. The judgment that separates a toy from a product: did the right memory get retrieved, did stale memory decay, and did the new turn get written back?

2. Mental Model

Explain Like I'm 12

Imagine a friend with a giant box of index cards. Every time something happens, they jot it on a card and toss it in the box: "Eric likes dark mode" (Tuesday), "we decided to use Groq" (last week), "brushed teeth" (this morning). The box is huge — they can't reread every card before answering you. So when you ask something, they grab the handful of cards that are (1) actually about your question, (2) recent or recently-used, and (3) a big deal — and ignore the rest. A card you keep pulling out stays near the top; a trivial old card sinks and fades. That three-part "which cards do I grab?" rule is the whole project.

Explain Like I'm a Software Engineer

• A memory stream is an append-only list of memory objects: {text, kind, created_at, last_accessed, importance, embedding}.
• The store lives in external context (out of the prompt — your "disk"); the prompt is main context ("RAM"), small and fixed. Retrieval is paging: select the top-k memories to load into the prompt for this turn.
• The ranking is a weighted sum of three signals: relevance = cosine(query, memory); recency = decay_rate ^ hours_since(last_accessed); importance = memory.importance / 10. Then score = w_rel·rel + w_rec·rec + w_imp·imp; sort descending, take k.
• Retrieving a memory refreshes its last_accessed, so used memories stay warm and unused ones decay. After the model answers, write the new turn back so the system grows.

Real-World Analogy

How It Works (Diagram)

 WRITE (every turn / observation):
   "Eric prefers Groq for the default API"  ──embed──► vector
                                              + created_at=now, last_accessed=now, importance=7, kind=semantic
                                              └─► append to memory stream (external context)

 RETRIEVE (per query, e.g. "which API should I default to?"):
   query ──embed──► q
        for each memory m in the stream:
            rel = cosine(q, m.embedding)                     # is it about this?   [0..1]
            rec = decay_rate ** hours_since(m.last_accessed) # is it fresh?        (0..1]
            imp = m.importance / 10                          # does it matter?     [0..1]
            score(m) = w_rel*rel + w_rec*rec + w_imp*imp
        top_k = sort(memories by score, desc)[:k]
        touch(top_k): last_accessed = now                    # retrieval refreshes recency
        inject top_k into the prompt ──► LLM answers grounded in memory

3. Technical Explanation

Formal Definition

A memory is an object m = (text, kind, created_at, last_accessed, importance, embedding); the memory stream is the list of all such objects. Retrieval scores every memory against a query q on three components and returns the top-k:

• relevance rel(q, m) = cos(e_q, e_m) — cosine similarity of the embeddings (Project 02).
• recency rec(m) = d^h, where d is the decay rate (the paper uses 0.995) and h = hours since m.last_accessed.
• importance imp(m) = importance / 10, a salience score rated 1–10 at creation.

Score: score(q, m) = w_rel·rel + w_rec·rec + w_imp·imp. The paper min-max normalizes each component to [0,1] and sets all weights to 1.

How It Works Step by Step

1. Write. When something happens, create a memory: embed its text once, stamp created_at = last_accessed = now, assign an importance (fixed default, or an LLM 1–10 rating), tag a kind, and append to the stream.
2. Score (relevance). Embed the query with the same model used for memories; relevance is the cosine similarity. Mixing models makes it meaningless — a silent bug.
3. Score (recency). rec = decay_rate ^ hours_since(last_accessed). At 0 hours this is 1.0. Recency is from last access, not creation — so retrieving and touching last_accessed keeps used memories warm.
4. Score (importance). imp = importance / 10. Set once at write time — "distinguishes mundane from core memories." It's why "we signed the contract" (10) outranks "brushed teeth" (1) at equal recency.
5. Combine & retrieve. score = w_rel·rel + w_rec·rec + w_imp·imp; sort descending; take the top-k that fit the budget. Touch the returned memories' last_accessed = now.
6. Inject & answer, then write back. Put the retrieved memories into the prompt (main context), call the LLM, then store the new turn as a memory so the system grows.

Key Concepts

4. Guided Examples

The lab stack: litellm (embeddings + chat via config.py), an in-memory MemoryStore, and pure-Python scoring (cosine, decay). Examples mirror the guiding tests.

Example 1: Simplest Case — exponential recency decay

from scoring import recency_score

now = 1_000_000.0           # epoch seconds (any fixed "now")
hour = 3600.0
# A memory accessed right now vs 1 hour ago vs 2 hours ago, decay_rate = 0.995:
print(recency_score(now, now,          decay_rate=0.995))   # 1.0      (0 hours)
print(recency_score(now, now - hour,   decay_rate=0.995))   # 0.995    (1 hour)
print(recency_score(now, now - 2*hour, decay_rate=0.995))   # 0.990025 (2 hours)

Example 2: Real-World Case — the three-signal score ranks memories

from scoring import retrieval_score

# Three already-computed (relevance, recency, importance) triples, equal weights:
w = (1.0, 1.0, 1.0)
relevant_but_old    = retrieval_score(rel=0.9, rec=0.10, imp=0.3, weights=w)  # 1.30
fresh_but_off_topic = retrieval_score(rel=0.1, rec=1.00, imp=0.2, weights=w)  # 1.30
relevant_and_fresh  = retrieval_score(rel=0.8, rec=0.90, imp=0.7, weights=w)  # 2.40  ← wins
print(relevant_and_fresh > relevant_but_old, relevant_and_fresh > fresh_but_off_topic)  # True True

Example 3: When It Fails — relevance-only retrieval grabs the wrong memory

from memory_store import Memory, MemoryStore
from retriever import retrieve

now = 1_000_000.0
store = MemoryStore()
# Same topic embedding [1,0,0]; one is trivial+old, one is important+fresh:
store.add(Memory("m0", "brushed teeth", kind="episodic", created_at=now-1e6,
                 last_accessed=now-1e6, importance=1, embedding=[1.0, 0.0, 0.0]))
store.add(Memory("m1", "decided to default to Groq", kind="semantic", created_at=now-10,
                 last_accessed=now-10,  importance=8, embedding=[1.0, 0.0, 0.0]))

q = [1.0, 0.0, 0.0]   # query embedding — equally relevant to BOTH memories
top = retrieve(store, q, now=now, k=1, weights=(1.0, 1.0, 1.0), decay_rate=0.995)
print(top[0].id)      # 'm1'  — recency + importance break the relevance tie correctly

5. Reflection Before Building

After filling in UNDERSTANDING.md, use the AI mentor pattern in docs/meta/learning-flow.md to get feedback. Record the feedback in UNDERSTANDING_FEEDBACK.md.

6. Project Assignment

See PROJECT.md and source/project.md for the full specification including file structure and extended requirements.

Core Requirement

Build a personal memory system in code/ that stores experiences and retrieves the most worth-remembering ones for a query, then a chat assistant that uses it:

• scoring.py — the three scoring signals and their combination: recency_score (exponential decay), importance_score (normalize 1–10), relevance_score (cosine), retrieval_score (weighted sum). Learner core — the learning target.
• retriever.py — score every memory in the store, sort, return the top-k, and touch their last_accessed. Learner core.
• memory_store.py — the Memory dataclass and an in-memory MemoryStore. (Provided — the data structure.)
• embedding_helpers.py — embeddings glue, same model both sides. (Provided — Project 02–04 reuse.)
• chat_with_memory.py — embed query → retrieve → inject memories → answer → write back. (Provided orchestrator.)

Extended: LLM-rated importance (poignancy 1–10); kind-aware decay (semantic/procedural resist decay); reflection (promote repeated episodic memories into durable facts); persistence across sessions.

7. Project Milestones

Work through these in order. Each milestone should produce runnable code before moving on.

8. Self-Evaluation

After building, honestly evaluate your implementation against these criteria. Record your answers in EVALUATION.md.

9. Common Mistakes

10. Connections

Builds On

Relevance scoring is Project 02's cosine similarity over embeddings, and retrieving top-k from a store is Project 03/04's retrieval — a memory system is that retrieval with two extra signals (recency, importance) stacked on top, ranking your own past instead of a PDF. The prompt that consumes the retrieved memories is Project 01's system+user message construction; the "same embedding model both sides" rule is inherited wholesale.

Enables

Project 08 (agent) turns write-back into a tool the model calls — MemGPT's self-editing memory. Project 09 (personal learning OS) builds on reflection: synthesizing durable semantic knowledge from accumulated episodic memories. Project 07 (evaluation) generalizes the LLM-rated importance into a full LLM-as-judge scorer.

Production Patterns

Real assistant-memory systems are this hierarchy plus engineering: a small in-prompt working memory and a large vector-backed store (MemGPT's main vs external context); retrieval scored on relevance + recency + importance (Generative Agents); episodic observations promoted to durable semantic facts via reflection; summarization on eviction when retrieval overflows; and the model itself deciding what to write via tools. Persistence (a real vector DB) and per-user isolation are the production add-ons.

StarcallOS Relevance

Sources

See source/resources.md for the complete annotated source list.