# Project 09: Personal Learning OS

# source/project.md — Detailed Project Specification

> Generated by: skills/lesson-generator
> For the high-level overview see the root PROJECT.md and the teaching content in
> `source/lesson.agent.md`.
> This file contains implementation contracts, not teaching content.

---

## Definition of Done

The project is complete when all of the following are true:

- [ ] Core implementation runs without errors (router classifies by precedence; `handle` routes → dispatches to one subsystem → returns a `Response` with route + provenance)
- [ ] All project milestones M1–M6 demonstrated (see `lesson.agent.md` §7)
- [ ] `route_query` orders checks by the cost of mistakes (`SAVE` ▸ `RECALL` ▸ `TASK` ▸ `CHAT`) and falls back to `CHAT` (the safe default) when no signal clears the threshold
- [ ] `KnowledgeGraph.add` links a new item to existing items **by shared tags, incrementally** (no full rebuild), with **no self-edge** and **symmetric** edges; `.related` ranks neighbors by shared-tag count
- [ ] `LearningOS.handle` dispatches to **exactly one** subsystem via the table, returns a `Response(answer, route, reason, provenance)`, and on `SAVE` links the new item into the graph
- [ ] `evaluate_routing` reports **overall** accuracy, **per-route** accuracy, and the explicit **misroute** list on a frozen labeled set
- [ ] UNDERSTANDING.md completed before first line of code
- [ ] FAILURE_ANALYSIS.md contains ≥3 intentional experiments, including the **"route everything to `TASK`"** (cost) and the **"mis-ordered precedence"** (lost save) ablations
- [ ] EVALUATION.md contains concrete results (overall + per-route routing accuracy, misroutes, and the cost/latency delta of the `TASK`-everything ablation), not impressions
- [ ] STARCALLOS_REFLECTION.md identifies at least one concrete applicable pattern
- [ ] Guiding tests in `code/tests/` pass (`python -m pytest`)

---

## File Specification

Seven files in `code/` (plus tests). The **subsystem kernel** (the in-memory store, the dispatch
table, the four worker callables) and the **app** are *provided* — they stand in for P01/P03/P05/P08,
which you already built. The friction here is the genuinely new work: the **router**, the
**knowledge graph**, the **orchestrator**, and the **system-level evaluation**.

### `config.py` — `provided`

Canonical provider-resolution block (do not edit) + a per-project `Config` adding `top_k` (5, recall
size), `route_threshold` (0.0, the confidence floor below which routing falls back to `CHAT`), and
`link_min_shared_tags` (1, the minimum shared tags to draw a graph edge). `temperature` defaults low
(0.2) for deterministic routing/synthesis. Reads `OS_TOP_K` / `OS_ROUTE_THRESHOLD` /
`OS_LINK_MIN_SHARED_TAGS` overrides.

### `subsystems.py` — `provided`

**Purpose:** The capability kernel — offline stand-ins for the prior projects, plus the dispatch table.

- `MemoryStore`: an in-memory list of `Item(id, text, tags, created_at)`; `save(text, tags) -> Item`
  and `recall(query, k) -> list[Item]` (keyword-overlap ranking — no network, so tests are offline).
- `SubsystemResult(answer: str, sources: list)`: what every worker returns (answer + the ids/sources used).
- The four workers, each `(query: str, ctx: Ctx) -> SubsystemResult`: `save_worker`, `recall_worker`,
  `task_worker` (a small bounded multi-recall+synthesis shim standing in for P08's agent), `chat_worker`
  (a single `chat(messages)` call).
- `make_subsystems(store, graph, chat) -> dict[str, Callable]`: builds the route→worker dispatch table,
  closing over the store, graph, and the injected `chat` function. Provided in full; re-read
  `projects/05-personal-memory-system` and `projects/08-ai-agent` if any worker is unfamiliar.

### `router.py` — `learner`

**Purpose:** The system's front door — classify a request into one route. (M1.)

**Provided:** the `Route` dataclass (`name`, `reason`, `confidence`) and the marker tables' *shape*
(`SAVE_MARKERS`, `RECALL_MARKERS`, `TASK_MARKERS` as starting lists you extend).

**Key function (learner-owned core):**

```python
def route_query(query: str, threshold: float = 0.0) -> Route:
    """Classify `query` into one of {SAVE, RECALL, TASK, CHAT}.
       Score each non-chat route by its matched markers; pick the highest-PRECEDENCE route
       (SAVE ▸ RECALL ▸ TASK) whose confidence >= threshold; else CHAT (the safe default).
       Return Route(name, reason=why, confidence)."""
```

**Example I/O:**
```text
route_query("remember that the demo is June 20").name   -> "SAVE"
route_query("what did I save about the demo?").name      -> "RECALL"
route_query("summarize everything I saved this week").name -> "TASK"
route_query("explain cosine similarity").name            -> "CHAT"   (no signal → safe default)
```

### `knowledge.py` — `partial`

**Purpose:** A lightweight personal knowledge graph over saved items. (M2.)

**Provided:** the `KnowledgeGraph` dataclass *fields* (`nodes: dict[id, set[tags]]`,
`edges: dict[id, dict[id, int]]`).

**Key methods (learner-owned core):**

```python
@dataclass
class KnowledgeGraph:
    nodes: dict = field(default_factory=dict)   # id -> set(tags)
    edges: dict = field(default_factory=dict)   # id -> {other_id: shared_tag_count}
    def add(self, item_id, tags, *, min_shared: int = 1) -> None:
        """Add the node and link it to EXISTING nodes sharing >= min_shared tags.
           Incremental (one pass over current members). No self-edge. Symmetric."""
    def related(self, item_id, k: int = 5) -> list:
        """Return up to k neighbor ids, sorted by shared-tag weight desc (never item_id itself)."""
```

**Example I/O:**
```text
g.add(1, {"ai","memory"}); g.add(2, {"ai","agents"}); g.add(3, {"cooking"})
g.related(1)   -> [2]          # shares "ai" with 2; nothing with 3
g.related(3)   -> []           # no shared tags
g.edges[1][2] == 1             # one shared tag ("ai"); symmetric: g.edges[2][1] == 1
```

### `learning_os.py` — `learner`

**Purpose:** The orchestrator — route, dispatch, synthesize with provenance. (M3.)

**Provided:** the `Response` dataclass (`answer`, `route`, `reason`, `provenance`).

**Key method (learner-owned core):**

```python
class LearningOS:
    def __init__(self, subsystems: dict, graph: KnowledgeGraph | None = None): ...
    def handle(self, query: str) -> Response:
        """route = route_query(query)
           result = self.subsystems[route.name](query)          # dispatch to EXACTLY ONE worker
           return Response(result.answer, route.name, route.reason, provenance=result.sources)."""
```

> `handle` must stay a pure router→dispatch→synthesize wrapper. It does **not** retrieve, save, or
> touch the graph itself — that would be the §8 red flag ("the orchestrator doing a worker's job").
> Each worker owns its capability: the provided `save` worker is what persists the item **and** links
> it into the knowledge graph. `handle` only routes, calls one worker, and attaches provenance.

**Example I/O:**
```text
os_.handle("remember StarcallOS routes requests").route        -> "SAVE"   (provenance=[new id])
os_.handle("what did I save about StarcallOS?").route          -> "RECALL" (provenance=[that id])
os_.handle("explain routing").route                            -> "CHAT"   (provenance=[])
# every Response carries route + provenance; handle dispatches to exactly one subsystem
```

### `evaluate_os.py` — `learner`

**Purpose:** Evaluate the router as a system — overall + per-route accuracy. (M4.)

**Key function (learner-owned core):**

```python
def evaluate_routing(cases: list[dict], route_fn) -> dict:
    """cases = [{"query": str, "expected_route": str}, ...]. Return:
       {"accuracy": correct/total,
        "per_route": {route: correct_r / total_r for each EXPECTED route present},
        "misroutes": [{"query","expected","got"} for each wrong case]}."""
```

**Example I/O:**
```text
evaluate_routing([{"query":"note: milk","expected_route":"SAVE"},
                  {"query":"summarize my week","expected_route":"TASK"}], route_query)
   -> {"accuracy": 0.5, "per_route": {"SAVE": 1.0, "TASK": 0.0},
       "misroutes": [{"query":"summarize my week","expected":"TASK","got":"RECALL"}]}
# overall hides the broken route; per_route exposes TASK at 0.0
```

### `os_app.py` — `provided`

**Purpose:** The orchestrator app. Builds a real LiteLLM-backed `chat(messages)`, wires
`make_subsystems(store, graph, chat)` into a `LearningOS`, and runs a REPL: for each line it prints the
chosen **route**, the **reason**, the **answer**, and the **provenance**. Calls the learner cores, so it
raises `NotImplementedError` until they're done — then runs end to end (M5).
`python os_app.py` (interactive) — or `python os_app.py "remember the demo is June 20"` (one-shot).

### `tests/` — `provided`

`test_router.py` (M1 route_query), `test_knowledge.py` (M2 graph), `test_learning_os.py`
(M3 orchestrator — fake subsystems + fake chat), `test_evaluate_os.py` (M4 evaluate_routing),
`test_config_models.py` (drift guard, passes today), `conftest.py` (puts `code/` on `sys.path`, plus
fake-subsystem / fake-chat helpers). All offline — no provider, no network.

---

## Input / Output Contracts

| Function | Input | Expected Output | Error Behavior |
|----------|-------|-----------------|----------------|
| `route_query(query, threshold)` | `str`, `float` | `Route(name ∈ {SAVE,RECALL,TASK,CHAT}, reason, confidence)` | empty/no-signal → `CHAT` (never raises) |
| `KnowledgeGraph.add(id, tags, min_shared)` | id, `set[str]`, `int` | mutates `nodes`/`edges`; links to sharers only | no shared tags → node with no edges |
| `KnowledgeGraph.related(id, k)` | id, `int` | `list` of up to `k` neighbor ids, weight desc | unknown id → `[]`; never includes `id` |
| `LearningOS.handle(query)` | `str` | `Response(answer, route, reason, provenance)` | dispatches one subsystem; always returns a `Response` |
| `evaluate_routing(cases, route_fn)` | `list[dict]`, fn | `dict(accuracy, per_route, misroutes)` | empty `cases` → accuracy `0.0`/safe, empty maps |
| `MemoryStore.save / .recall` | text+tags / query+k | `Item` / ranked `list[Item]` | provided (carried from P05) |

---

## Extended Requirements

Beyond the core implementation — complete these after the core is working.

- [ ] **LLM-backed router:** `route_query_llm` via a single low-temp model call returning a route label; compare it to the deterministic router on the same frozen set with `evaluate_routing` (source: `sources/articles/building-effective-agents.md`).
- [ ] **Provenance into the prompt:** pass retrieved items to the model on `RECALL`/`TASK` and have it cite them inline, RAG-style (sources: `sources/papers/rag-paper.md`, `sources/official-docs/anthropic-citations.md`).
- [ ] **Graph-aware recall:** expand recall hits with `graph.related(id)` so connected notes surface without a direct keyword match (source: `sources/papers/generative-agents.md`).
- [ ] **Reflection job:** a periodic step that summarizes recent items into a new higher-level item + graph node (source: `sources/papers/generative-agents.md`).
- [ ] **Threshold tuning:** sweep `route_threshold`; plot routing precision/recall; pick the value that minimizes dangerous misroutes.

---

## Known Difficulty Spikes

Listed so the learner expects them, not so they can avoid them.

1. **Precedence is correctness.** The order of route checks encodes the asymmetric cost of mistakes. Checking `CHAT` first silently drops saves; the order `SAVE ▸ RECALL ▸ TASK ▸ CHAT` is the design decision, not the markers.
2. **The safe default.** When no signal clears the threshold, fall to `CHAT` — never `TASK` (cost) or `SAVE` (junk). Getting the *fallback* wrong is as damaging as getting a positive match wrong.
3. **Keep `handle` thin.** The orchestrator routes and synthesizes; it must **not** retrieve or save itself. If `handle` grows worker logic, you've lost the separation of concerns routing buys.
4. **Provenance is returned, not logged.** Put `route`/`reason`/`provenance` in the `Response`. A `print` is invisible to the tests and the evaluator.
5. **Graph: incremental, symmetric, no self-edge.** Link the new node to current members in one pass; mirror the edge weight both ways; never link a node to itself. Rebuilding all pairs is O(N²).
6. **Per-route, not just the mean.** `evaluate_routing` must expose a route that's silently 0%. An overall number alone is the metric trap this project is partly about.

---

## Debugging Approach

When things break, check in this order:

1. Environment — is `.env` loaded? (Only `os_app.py`'s live run needs a provider; all tests run offline.)
2. Router — `python -m pytest tests/test_router.py`: do save/recall/task phrasings route correctly, and does an unsignaled request fall to `CHAT`? Check **precedence** (a "remember…" must not become `CHAT`).
3. Graph — `python -m pytest tests/test_knowledge.py`: are edges symmetric, self-free, and incremental? Does `related` rank by shared-tag weight?
4. Orchestrator — `python -m pytest tests/test_learning_os.py`: does `handle` dispatch to exactly one subsystem, round-trip a save→recall, and carry provenance on every `Response`?
5. Evaluation — `python -m pytest tests/test_evaluate_os.py`: does a deliberately broken route show as 0% per-route while overall stays high?
6. Live — `python os_app.py`: type "remember the demo is June 20", then "what did I save about the demo?", then "summarize everything I saved" — watch the printed route + provenance change.
7. Source — re-read `source/lesson.agent.md` §3 and `sources/articles/building-effective-agents.md` (routing + orchestrator-workers; "simplest thing that works").

---

## Integration Notes

**Depends on:** Project 05 (the memory store — `RECALL`/`SAVE` route to it; provided here), Project 03
(retrieval ranking behind recall), Project 08 (the reliable agent behind the `TASK` route; provided as a
bounded shim), Project 04 (citation/provenance — the orchestrator's `Response` carries it), Project 07
(evaluation discipline — `evaluate_routing` is per-case scoring applied to the router), Project 01 (the
single call behind `CHAT`).

**Depended on by:** This is the capstone — nothing in the core curriculum depends on it. It is the
direct prototype for **StarcallOS**: one conversational front door, a router, a capability/dispatch
registry, provenance on every answer, and a system-level evaluation of the front door itself.