Personal Learning OS

Learning Objectives

By the end of this capstone, you will be able to:

• Describe a multi-capability AI app as an augmented LLM assembled into a system (retrieval + tools + memory), and see the eight prior projects as subsystems of one OS.
• Implement query routing — classify a request and "direct it to a specialized followup task" — and explain why that buys "separation of concerns."
• Justify a routing precedence and a safe default: order the checks so a high-stakes intent (save) is never shadowed by a generic one, and fall back to the cheapest safe route.
• Build an orchestrator that routes, dispatches to one subsystem, and synthesizes a response with provenance — which route handled it and which items it used.
• Build a lightweight personal knowledge graph that links saved items by shared tags and surfaces related ones.
• Evaluate the system, not just a part: measure routing accuracy overall and per route, and explain why a mean hides a silently-broken route.
• Apply "use the simplest thing that works" at the system level: route a simple lookup to a single retrieval, reserve the agent for the genuinely open-ended request.
• Name the major failure modes of an orchestrated system (a dropped save, everything-to-the-agent, no provenance, a router that regresses unseen) and the decision that prevents each.

1. Motivation

Why This Exists

By Project 08 you have eight working capabilities: a chatbot (P01), embeddings (P02), semantic search (P03), retrieval with citations (P04), a memory system (P05), a tool-using copilot (P06), an evaluation harness (P07), and a reliable autonomous agent (P08). Each runs on its own. But a person doesn't think in projects — they say "remember this," "what did I note about X," "summarize everything I've saved on Y," or just "explain this to me." A Personal Learning OS is the front door that takes any of those and sends it to the right capability. The new problem is no longer how to build a retriever or an agent — you've done that. It's how to compose them.

What Breaks Without It

The cost of routing wrong is concrete: mis-route a "remember my doctor's appointment" to small-talk chat and the note is silently lost; mis-route a one-line lookup to the agent and you've spent ten steps and real money on something a single retrieval would have answered. A system with brilliant specialists and a bad front door feels broken even when every individual subsystem works perfectly.

Real-World Stakes

This is the shape of every "AI assistant" product: a single conversational surface backed by many specialized capabilities. ChatGPT routes between chat, browsing, code execution, and image generation. A personal-knowledge tool routes between capture, recall, and synthesis. The product is not any single capability — those are commodities. The product is the orchestration: the right specialist for the request, every time, with a trail you can inspect.

2. Mental Model

Explain Like I'm 12

Imagine a big help desk with one friendly person at the front and four expert rooms behind them. You walk up and say anything. The front-desk person's only job is to figure out which room you need: the Filing Room if you're handing them something to keep ("remember this"), the Library if you're asking for something you stored before ("what did I write about my trip?"), the Workshop for a big multi-step job ("go through all my notes about France and make me a summary"), or the Chat Couch if you just want to talk. If they guess wrong, bad things happen: hand a note to the Couch and it gets thrown away; send a one-second question to the Workshop and you wait an hour for what the Library could've handed you instantly. So they follow simple rules — "if they're handing me something to keep, that's always the Filing Room first" — and when they truly can't tell, they default to the Couch, which is safe and cheap. This project is you building that front desk.

Explain Like I'm a Software Engineer

• The OS is the augmented LLM as a system: retrieval (P03/P05), tools/agent (P06/P08), and memory (P05), wired behind one entrypoint. The subsystems are provided (you built them); the learning target is the composition.
• route_query(query) -> Route is a classifier. Deterministic and signal-based so it's offline-testable: match intent markers, score them, return the highest-precedence route above a confidence threshold, else the safe default (CHAT). Production routing "can be handled by an LLM or a more traditional classification model."
• Precedence is correctness, not style. SAVE is checked before CHAT because a dropped save is a data-loss failure; TASK (the agent) is gated so only genuinely multi-step requests reach it.
• LearningOS.handle(query) is the orchestrator: route → look up the subsystem in a dispatch table → call it → wrap the result in a Response carrying route, reason, and provenance.
• KnowledgeGraph links saved items by shared tags; related(id) returns neighbors ranked by shared-tag count — the reflection idea at engineering scale.
• evaluate_routing(cases, route_fn) scores the router on a frozen labeled set: overall accuracy and per-route accuracy, plus the misroutes.

Real-World Analogy

How It Works (Diagram)

 REQUEST: "remember that my StarcallOS demo is on June 20"
   │  route_query → matches SAVE markers ("remember that") → Route(SAVE)
   ▼  handle → store.save(item, tags=[...]) → graph.add(id, tags)
   ▼  Response(answer="Saved.", route="SAVE", provenance=[item#42])

 REQUEST: "what did I save about StarcallOS?"
   │  route_query → matches RECALL markers ("what did I save") → Route(RECALL)
   ▼  handle → store.recall("StarcallOS") → [#42, #17]
   ▼  Response(answer="You noted: demo on June 20; …", route="RECALL", provenance=[#42,#17])

 REQUEST: "go through everything I know about StarcallOS and draft a status summary"
   ▼  Route(TASK) → bounded agent over the store (P08) → multi-step synthesis

 REQUEST: "what's the difference between recall and precision?"   (no personal data)
   ▼  no strong signal → Route(CHAT, safe default) → single LLM call

 evaluate_routing(labeled_cases, route_query)
   ─► {accuracy: 0.92, per_route: {SAVE:1.0, RECALL:0.9, TASK:0.8, CHAT:0.95}, misroutes:[…]}

3. Technical Explanation

Formal Definition

Routing is the workflow where the system "classifies an input and directs it to a specialized followup task." It "allows for separation of concerns, and building more specialized prompts." A route is one of a fixed, small set {SAVE, RECALL, TASK, CHAT}, each bound to exactly one subsystem; route_query returns the chosen route plus a reason and a confidence. The orchestrator follows the orchestrator-workers pattern: a central component that "dynamically breaks down tasks, delegates them to worker [subsystems], and synthesizes their results." Provenance is the set of item ids / source pointers that produced the answer. A knowledge graph here is an undirected weighted graph where an edge's weight is the number of tags two saved items share.

How It Works Step by Step

1. Classify the request (routing). route_query scans the text for intent markers per route, scores each, and selects the highest-precedence route whose confidence clears the threshold; if none does, return the safe default CHAT. Precedence — SAVE ▸ RECALL ▸ TASK ▸ CHAT — ranks routes by the cost of mis-routing.
2. Dispatch to one subsystem. handle looks the route up in a dispatch table and calls exactly one subsystem. One request → one specialist — the "separation of concerns" routing buys.
3. Synthesize with provenance. Each subsystem returns a SubsystemResult(answer, sources). The orchestrator wraps it into a Response(answer, route, reason, provenance=sources) — what makes the system auditable.
4. Grow the graph on save. When the route is SAVE, the save worker records the item and links it into the graph by shared tags. Later, recall can surface connected notes.
5. Evaluate the router. evaluate_routing runs over a frozen labeled set and computes overall accuracy, per-route accuracy, and the explicit misroute list — the only way to catch a route that's silently at 0%.

Key Concepts

4. Guided Examples

The lab stack: the provided subsystem kernel (MemoryStore, the dispatch table, an injected chat), a deterministic signal-based router, a tiny pure-Python graph, and a pure-logic evaluator. Examples mirror the guiding tests; all run offline.

Example 1: Simplest Case — classify a request

from router import route_query

route_query("remember that my StarcallOS demo is on June 20").name   # -> "SAVE"
route_query("what did I save about StarcallOS?").name                # -> "RECALL"
route_query("go through all my notes on France and draft a summary").name  # -> "TASK"
route_query("explain the difference between recall and precision").name    # -> "CHAT"  (safe default)

r = route_query("remember to email Sam")
print(r.name, "|", r.reason)   # SAVE | matched SAVE markers: ['remember']

Example 2: Real-World Case — orchestrate a request end to end

from learning_os import LearningOS
from subsystems import MemoryStore, make_subsystems
from knowledge import KnowledgeGraph

store, graph = MemoryStore(), KnowledgeGraph()
# fake chat fn so this is offline; the real app injects a LiteLLM-backed one
os_ = LearningOS(make_subsystems(store, graph, chat=lambda msgs: "(chat answer)"), graph)

r1 = os_.handle("remember that StarcallOS uses a routing front door")   # SAVE
print(r1.route, r1.provenance)        # 'SAVE' [<id of the new item>]

r2 = os_.handle("what did I save about StarcallOS?")                    # RECALL
print(r2.route, r2.provenance)        # 'RECALL' [<id of item saved above>]

Example 3: When It Fails — the per-route number a mean hides

from evaluate_os import evaluate_routing
from router import route_query

cases = [
    {"query": "remember to call mom", "expected_route": "SAVE"},
    {"query": "note: buy milk", "expected_route": "SAVE"},
    {"query": "what did I note about milk?", "expected_route": "RECALL"},
    {"query": "summarize everything I saved this week", "expected_route": "TASK"},
    {"query": "what is a vector database?", "expected_route": "CHAT"},
]
report = evaluate_routing(cases, route_query)
print(report["accuracy"])     # e.g. 0.8  (overall — looks fine)
print(report["per_route"])    # e.g. {'SAVE': 1.0, 'RECALL': 1.0, 'TASK': 0.0, 'CHAT': 1.0}
print(report["misroutes"])    # [{'query': 'summarize everything I saved this week', ...}]

5. Reflection Before Building

After filling in UNDERSTANDING.md, use the AI mentor pattern in docs/meta/learning-flow.md to get feedback. Record it in UNDERSTANDING_FEEDBACK.md.

6. Project Assignment

See PROJECT.md and source/project.md for the full specification including file structure and extended requirements.

Core Requirement

Build the composition layer that turns eight separate capabilities into one Personal Learning OS, in code/:

• router.py — route_query (M1) is the learner core: the deterministic, precedence-ordered, safe-default classifier that is the system's front door.
• knowledge.py — KnowledgeGraph.add / .related (M2) is the learner core: link saved items by shared tags and surface related ones, incrementally.
• learning_os.py — LearningOS.handle (M3) is the learner core: the orchestrator — route → dispatch to one subsystem → synthesize a Response with provenance.
• evaluate_os.py — evaluate_routing (M4) is the learner core: score the router overall and per route on a frozen set, and list the misroutes.
• subsystems.py — the provided kernel: MemoryStore, make_subsystems, and the four workers. (Provided — they stand in for P01/P03/P05/P08.)
• os_app.py — the orchestrator app: a real LiteLLM chat, the subsystems, a REPL that prints route + reason + answer + provenance. (Provided.)

Extended: an LLM-backed router compared against the deterministic one with evaluate_routing; provenance passed into the prompt for inline citations; graph-aware recall that expands hits with related(id); a reflection job that summarizes recent items; threshold tuning for routing precision/recall.

7. Project Milestones

Work through these in order. Each milestone should produce runnable code before moving on.

8. Self-Evaluation

After building, honestly evaluate your implementation against these criteria. Record your answers in EVALUATION.md.

9. Common Mistakes

10. Connections

Builds On

This is the whole course assembled into one system. CHAT is Project 01 (a single completion). RECALL is Projects 03 and 05 (retrieval over a memory store). SAVE is Project 05 plus the new knowledge graph. TASK is Project 08's reliable agent (and through it, Project 06's tools). The provenance the orchestrator attaches is Project 04's citation discipline. The evaluation of the router is Project 07/08's "numbers, per case" discipline pointed at the front door. Nothing here re-teaches a capability — every subsystem is provided precisely because you already built it. The new work is the one thing the prior projects deliberately left out: how to compose them.

Enables

This is the terminal project of the core curriculum — but it's the first project of building a real product. Everything beyond here is depth on a subsystem (better retrieval, a smarter router, more tools, richer evaluation) or breadth (more routes, more capabilities behind the same front door). The pattern you build — one entrypoint, a router, a dispatch table, provenance, and a system-level eval — is the skeleton every multi-capability AI app grows on.

One concrete next step is the MCP Interface Layer elective. Here the subsystems behind route_query are wired in-process; the elective shows how each (memory, retrieval, the agent) could instead be re-exposed as a Model Context Protocol server and the OS could consume them — and third-party servers — through one client. That turns the dispatch table from a hard-coded map into a registry of interchangeable, separately-deployable capabilities: the provider/consumer split applied to the very system you just composed. Routing and orchestration stay here in the host; the capabilities move behind a standard protocol surface.

Production Patterns

Real assistant systems are this skeleton plus production engineering: a router (rules, a small classifier, or an LLM) with a confidence threshold and a safe fallback; a capability/dispatch registry; per-capability prompts and budgets (P08's reliability layer on the TASK route); provenance and citations on every answer (P04); a frozen evaluation set for the router and each subsystem (P07); and observability — a trace of which route handled what. Anthropic's guidance frames the spectrum: a single augmented LLM call, then workflows (prompt chaining, routing, orchestrator-workers, evaluator-optimizer), then a full agent — reach for the simplest that works, and compose the named patterns rather than building one monolith.

StarcallOS Relevance

Sources

See source/resources.md for the complete annotated source list.