# Project 01: AI Chatbot

# source/project.md — Detailed Project Specification

> Generated by: skills/lesson-generator
> For the high-level overview see the root PROJECT.md.
> For the teaching content see source/lesson.agent.md.
> This file contains implementation contracts, not teaching content.

---

## Definition of Done

The project is complete when all of the following are true:

- [ ] Core implementation runs without errors against a real provider
- [ ] All five milestones (M1–M5 in lesson.agent.md §7) demonstrated
- [ ] Multi-turn history is correct: BOTH user and assistant turns appended every turn [src: anthropic-messages-api]
- [ ] A configurable system prompt visibly changes behavior (same question, two prompts → two styles)
- [ ] Responses stream token-by-token, not all at once [src: anthropic-streaming]
- [ ] Per-turn and cumulative cost are derived from real `usage`, and match a hand calculation for one known turn [src: anthropic-pricing]
- [ ] UNDERSTANDING.md completed before the first line of code
- [ ] FAILURE_ANALYSIS.md contains at least 3 intentional experiments (see "Intentional Breakage Menu" below)
- [ ] EVALUATION.md contains quantitative results (real token/cost numbers), not impressions
- [ ] STARCALLOS_REFLECTION.md identifies at least one concrete applicable pattern

---

## Architecture Overview

A single-process CLI loop. No server, no database, no async UI framework — the conversation state lives in memory as a Python list and is the entire "memory" of the bot. [src: anthropic-messages-api]

```
        ┌─────────────────────────────────────────────┐
        │                 chatbot.py                   │
        │  REPL loop: read → build request → call →    │
        │  stream → append → report cost → repeat      │
        └───────┬───────────────┬──────────────┬───────┘
                │               │              │
          config.py        context.py     cost_tracker.py
        (model, temp,    (token estimate,  (usage → $,
         max_tokens,      window guard,     per-turn +
         system prompt)   trim/summarize)   cumulative)
                │               │              │
                └───────────────┴──────────────┘
                                │
                         litellm.completion()
                     (provider-agnostic transport) [src: litellm-completion]
```

**Data flow per turn:** user input → append `{"role":"user",...}` → `context.guard(history)` → `litellm.completion(stream=True)` → accumulate streamed deltas → append `{"role":"assistant",...}` → `cost_tracker.record(usage)` → print cost.

---

## File Specification

> **File roles** (per `OPERATING_RULES.md` §Scaffolding Rules): `provided` = complete support code; `partial` = starter with `TODO(learner)`/`NotImplementedError`; `learner` = you write the core; `reference` = docs/rubric. Setup is solved and interfaces are given — the learning target is left incomplete. Starter files live in `code/`.

### config.py — `provided`

**Purpose:** Single source of truth for all tunable knobs. No logic that touches the API — just values and the loaded environment.

**Key functions / contract:**

```python
from dataclasses import dataclass

@dataclass(frozen=True)
class Config:
    model: str = "claude-sonnet-4-6"      # any LiteLLM model string [src: litellm-completion]
    temperature: float = 0.7              # 0.0 deterministic-ish … 1.0 creative [src: anthropic-messages-api]
    max_tokens: int = 1024                # caps OUTPUT of one call only [src: anthropic-messages-api]
    context_budget: int = 100_000         # soft token ceiling for history (model-dependent)
    system_prompt: str = "You are a concise, helpful assistant."

def load_config() -> Config:
    """Read overrides from env / .env, return a frozen Config. Loads API keys into env as a side effect."""
```

**Dependencies:** `python-dotenv` (load `.env`), `dataclasses`.

**Does not:** call the API, count tokens, or hold conversation state.

---

### cost_tracker.py — `partial`

**Purpose:** Convert `usage` token counts into dollars and accumulate spend across the session. Starter has signatures + `PRICES`; you implement `cost_of()` and `record()`. Guiding tests in `code/tests/` must pass.

**Key functions / contract:**

```python
# ($/MTok input, $/MTok output) — values from sources/official-docs/anthropic-pricing.md
PRICES: dict[str, tuple[float, float]] = {
    "claude-sonnet-4-6": (3.0, 15.0),
    "claude-haiku-4-5":  (1.0,  5.0),
    # extend as needed; missing model → raise, do not silently guess
}

def cost_of(model: str, input_tokens: int, output_tokens: int) -> float:
    """cost = in/1e6 * in_price + out/1e6 * out_price. Raises KeyError on unknown model."""

class CostTracker:
    def record(self, model: str, input_tokens: int, output_tokens: int) -> float:
        """Add one turn; return that turn's cost. Maintains cumulative total + token totals."""
    @property
    def total(self) -> float: ...
    def summary(self) -> str:
        """Human-readable: turns, total input/output tokens, cumulative $."""
```

**Dependencies:** none (pure Python). This is what makes it unit-testable without the network.

**Does not:** call the API or estimate tokens for un-sent text (that's `context.py`). Cost is computed from *actual* returned `usage`, never guessed. [src: anthropic-pricing]

---

### context.py — `partial`

**Purpose:** Keep the `messages` array from exceeding the context window. Estimate size *before* sending; trim or warn when over budget. [src: anthropic-messages-api] Starter has the three signatures; you implement the bodies — `trim_to_budget()` is the real design decision. Offline guiding tests in `code/tests/test_context.py` (no network) must pass; they assert `trim_to_budget` by property (system message kept, result within budget, newest turns survive, caller's list not mutated), so your dropping strategy has room.

**Key functions / contract:**

```python
def estimate_tokens(messages: list[dict], system: str = "") -> int:
    """Rough estimate: ~4 chars/token over all content + system. [src: anthropic-pricing]
    Estimate only — the true count comes back in usage.input_tokens after the call."""

def within_budget(messages: list[dict], budget: int, system: str = "") -> bool:
    """True if estimated tokens <= budget."""

def trim_to_budget(messages: list[dict], budget: int, system: str = "") -> list[dict]:
    """Drop OLDEST user/assistant turns (sliding window) until within budget.
    Never drops the system message. Preserves role alternation. Returns a new list."""
```

**Dependencies:** none required for the core; optionally a real tokenizer for accuracy (note the tradeoff in IMPLEMENTATION.md).

**Does not:** call the API, decide pricing, or mutate the caller's list in place (return a new list).

**Extended:** a `summarize_old_turns(...)` variant that compresses dropped turns into one synthetic system note instead of discarding them. [src: anthropic-pricing]

---

### chatbot.py — `learner` (the main learning target)

**Purpose:** The orchestrator — the REPL that wires config + context + cost_tracker around `litellm.completion()`. Starter provides command dispatch and module wiring; **you** build `stream_completion()` and the conversation-turn body (build request → stream → append BOTH roles → record cost). This is the heart of the project — do not look for it pre-written.

**Key functions / contract:**

```python
def stream_completion(history: list[dict], cfg: Config) -> tuple[str, dict]:
    """Call litellm.completion(stream=True); print deltas live; return (full_text, usage_dict).
    usage may be absent/zero on some providers when streaming — fall back to a non-stream
    usage call or a local estimate, and say which in IMPLEMENTATION.md. [src: anthropic-streaming]"""

def run_repl(cfg: Config) -> None:
    """Main loop:
      1. seed history with the system message (LiteLLM/OpenAI shape) [src: litellm-completion]
      2. read user input; handle slash commands (/cost, /reset, /system, /quit)
      3. append user turn; trim_to_budget; call stream_completion
      4. append assistant turn; record + print cost
    """

def main() -> None:
    """load_config() → run_repl(). Entry point."""
```

**Slash commands (minimum):** `/cost` (print `CostTracker.summary()`), `/reset` (clear history back to just the system message), `/system <text>` (swap the system prompt live), `/quit`.

**Dependencies:** `config.py`, `context.py`, `cost_tracker.py`, `litellm`.

**Does not:** hardcode prices, hardcode the model, or send only the latest message (must resend full history). [src: anthropic-messages-api]

---

## Input / Output Contracts

> The observable behavior the implementation must satisfy. These are the testable acceptance criteria.

| Function / Behavior | Input | Expected Output | Error Behavior |
|--------------------|-------|-----------------|----------------|
| `cost_of` | `("claude-sonnet-4-6", 1_000_000, 0)` | `3.0` | unknown model → `KeyError` (no silent 0) |
| `cost_of` | `("claude-sonnet-4-6", 0, 1_000_000)` | `15.0` | — |
| Multi-turn memory | Turn 1 "Capital of France?" → "Paris"; Turn 2 "Its population?" | Turn 2 answer references Paris (pronoun resolved) | If only latest msg sent → bot can't resolve "its" |
| `input_tokens` growth | Same question asked at turn 2 vs turn 50 | `usage.input_tokens` strictly larger at turn 50 | — |
| System prompt effect | Same question under "answer in one word" vs "answer verbosely" | Visibly different response length/style | — |
| Streaming | any prompt | Text prints incrementally (multiple writes), not one block | stream error → surface it, don't hang |
| `trim_to_budget` | history estimated > budget | returned list estimated <= budget, system msg retained, roles still alternate | budget smaller than system msg → keep system msg, warn |
| `/reset` | issued mid-conversation | history == `[system message]`; next turn has no memory of prior turns | — |

---

## Extended Requirements

Beyond the core CLI — complete after the core is working and demonstrated.

- [ ] **Context-window guard:** warn (or auto-trim) before a call would exceed `context_budget`; show estimated vs actual tokens. [src: anthropic-messages-api]
- [ ] **Provider switch:** change one config value (`model`) to run the identical loop on a different provider/model and confirm no other code changes. [src: litellm-completion]
- [ ] **Transcript persistence:** write the conversation (+ per-turn cost) to a JSON/markdown file on `/quit`; load it back to resume.
- [ ] **`stop_reason` handling:** detect `max_tokens` truncation and tell the user the answer was cut off. [src: anthropic-messages-api]
- [ ] **Summarization trim:** replace sliding-window dropping with summarize-old-turns and compare cost/quality.

---

## Known Difficulty Spikes

Listed so the learner expects them, not so they can avoid them.

1. **Streaming + usage is the hardest part.** Streamed chunks frequently do **not** carry final `usage`, so naive cost tracking reports `$0.00`. Expect to either request usage explicitly, use a final-message call, or fall back to a local token estimate — and to document which. [src: anthropic-streaming]
2. **The "forgotten append" bug feels like a model problem.** When the assistant reply isn't appended to history, the bot acts amnesiac and learners blame the model. The bug is in the loop, not the API. [src: anthropic-messages-api]
3. **System prompt placement differs by interface.** Raw Anthropic API uses a top-level `system` field; LiteLLM/OpenAI shape expects a `{"role":"system"}` message at index 0. Mixing the two silently drops the prompt. [src: anthropic-messages-api] [src: litellm-completion]
4. **Token estimates lie.** `~4 chars/token` is good enough to guard a budget but will not match `usage.input_tokens` exactly — learners often expect equality and chase a non-bug. [src: anthropic-pricing]
5. **Role alternation after trimming.** Sliding-window trimming can accidentally leave two consecutive same-role messages, which providers merge or reject — trim in user/assistant *pairs*. [src: anthropic-messages-api]

---

## Intentional Breakage Menu

> For FAILURE_ANALYSIS.md — pick at least 3, break it on purpose, diagnose precisely.

- Send only the latest user message (drop history) → observe amnesia; tie it to statelessness.
- Stop appending the assistant reply → identify the exact turn the thread breaks.
- Set `max_tokens=10` and ask for a long answer → catch `stop_reason == "max_tokens"`.
- Hardcode the cost instead of reading `usage` → show how far off the estimate drifts over a long chat.
- Remove the context guard and run 100+ turns → reach the window limit; record the error.
- Put instructions in a first user message instead of `system` → watch them get diluted as the chat grows.

---

## Debugging Approach

When things break, check in this order:

1. **Environment** — is `.env` loaded? Is the API key valid and for the right provider?
2. **Inputs** — log the full `messages` array (and `system`) just before the call. Is the history actually there?
3. **Outputs** — log the raw response/`usage` before parsing. Is `usage` present at all?
4. **Isolation** — does `cost_of` pass its unit cases with no network? Does `trim_to_budget` shrink a synthetic oversized history?
5. **Source** — re-read the relevant section of `source/lesson.agent.md` (§3 mechanics, §9 common mistakes).

---

## Integration Notes

**Depends on:** none — this is Project 1, the foundation. [src: mlabonne-llm-course]

**Depended on by:**
- **P2 (Tokens & Embeddings):** deepens `estimate_tokens` into real tokenization.
- **P4 (RAG):** injects retrieved docs into the same `messages` array, making the context-window guard here load-bearing.
- **P5 (Memory):** replaces unbounded history resend with summarize/store — the direct answer to difficulty spike #5 and the "history grows unbounded" mistake.
- **P8 (Agent):** the REPL becomes a tool-use loop where `stop_reason == "tool_use"` replaces `end_turn`. [src: anthropic-messages-api]

**StarcallOS:** the config + cost_tracker + context guard triplet is the reusable substrate for any assistant surface; the quadratic input-token cost is the concrete motivation for StarcallOS's persistent memory layer.