# Project 08: AI Agent

# source/project.md — Detailed Project Specification

> Generated by: skills/lesson-generator
> For the high-level overview see the root PROJECT.md and the teaching content in
> `source/lesson.agent.md`.
> This file contains implementation contracts, not teaching content.

---

## Definition of Done

The project is complete when all of the following are true:

- [ ] Core implementation runs without errors (loop: call → dispatch **with recovery** → feed results back → guard → repeat; returns a `RunResult`)
- [ ] All project milestones M1–M6 demonstrated (see `lesson.agent.md` §7)
- [ ] `detect_stuck` returns `True` only when the last `window` actions are **identical** (tool **and** arguments), and `False` for different productive steps
- [ ] `BudgetTracker` enforces a **steps** ceiling and a **tokens** ceiling; `over_budget()` returns a *reason string* when crossed and `None` otherwise
- [ ] `run_agent` catches a **raising** tool, feeds the error back as an observation, and keeps looping (recovery) — and appends the assistant turn **before** the tool results
- [ ] `run_agent` returns a distinct `stop_reason` ∈ {`answered`, `stuck`, `budget`, `max_steps`} — failures are never collapsed into success
- [ ] `evaluate_run` reports `completed`, `steps`, `tool_calls`, and `efficiency` for a run
- [ ] UNDERSTANDING.md completed before first line of code
- [ ] FAILURE_ANALYSIS.md contains ≥3 intentional experiments, including the **"agent where a single call would do"** ablation
- [ ] EVALUATION.md contains concrete results (stop reasons, step/tool-call counts, efficiency, token cost), not impressions
- [ ] STARCALLOS_REFLECTION.md identifies at least one concrete applicable pattern
- [ ] Guiding tests in `code/tests/` pass (`python -m pytest`)

---

## File Specification

Seven files in `code/` (plus tests). The chat call, the **entire P06 tool layer** (sandbox, schemas,
dispatch, parsing), the `RunResult`/`Action` records, the token estimator, and the orchestrator are
**provided** — because you built the tool loop in Project 06. The friction here is the genuinely new
work: the **reliability layer** (stuck detection, budget, recovery) and **evaluating a run**.

### `config.py` — `provided`

Canonical provider-resolution block (do not edit) + a per-project `Config` adding `repo_root` (`"."`),
`max_steps` (10), `token_budget` (20000), and `temperature` defaulting low (0.2) for deterministic
agent behavior. Reads `AGENT_REPO_ROOT` / `AGENT_MAX_STEPS` / `AGENT_TOKEN_BUDGET` overrides.

### `tools.py` — `provided`

**Purpose:** The agent's action space + its security boundary — **the complete Project 06 tool layer**,
already implemented. `safe_resolve` (path sandbox), `read_file` / `list_directory` / `search_code`
(sandboxed file ops), `dispatch_tool` (router), and `TOOL_SCHEMAS` (the ACI). Provided in full so
effort goes to the control layer; re-read `projects/06-ai-coding-copilot` if any of it is unfamiliar.

### `safety.py` — `partial`

**Purpose:** The reliability primitives. (M1, M2.)

**Provided:** the `Action` dataclass (`tool`, `arguments`) and the `BudgetTracker` dataclass *fields*.

**Key functions (learner-owned core):**

```python
def detect_stuck(history: list[Action], window: int = 3) -> bool:
    """True iff the last `window` actions are all IDENTICAL (same tool AND arguments) — no progress.
    < window actions -> False. Compare whole actions, not just the tool name."""

@dataclass
class BudgetTracker:
    max_steps: int
    max_tokens: int
    steps: int = 0
    tokens: int = 0
    def tick(self, tokens_used: int) -> None:
        """Record one step: steps += 1; tokens += tokens_used."""
    def over_budget(self) -> str | None:
        """Return a reason string if steps OR tokens crossed their ceiling; else None."""
```

**Example I/O:**
```text
detect_stuck([read a, search foo, search foo, search foo], 3)  -> True
detect_stuck([search x, read a, read b], 3)                    -> False
BudgetTracker(5, 10_000); tick(2000); tick(2000); over_budget()  -> None
  then tick(7000); over_budget()  -> "token budget exhausted (11000/10000)"
```

### `agent.py` — `learner`

**Purpose:** The core — the generalized, reliable agent loop. (M3.)

**Provided:** the `RunResult` dataclass; `parse_tool_calls` (carried from P06, implemented);
`estimate_tokens` (offline token estimate). `dispatch_tool` / `TOOL_SCHEMAS` imported from `tools.py`.

**Key function (learner-owned core):**

```python
def run_agent(messages: list, repo_root, complete, budget: BudgetTracker, *,
              tools: list = TOOL_SCHEMAS, stuck_window: int = 3) -> RunResult:
    """Loop (structurally bounded by budget.max_steps):
       msg = complete(messages, tools); budget.tick(estimate_tokens(msg))
       calls = parse_tool_calls(msg)
       if no calls -> RunResult(answer=msg.content, ..., stop_reason="answered")
       append msg; for each call: record Action; result = dispatch_tool(...) INSIDE try/except
         (on Exception -> result = f"Error: {e}"  <- RECOVERY); append the tool-result message.
       if detect_stuck(history, stuck_window) -> RunResult(..., stop_reason="stuck")
       if budget.over_budget() -> RunResult(..., stop_reason="budget")
       loop exhausted -> RunResult(..., stop_reason="max_steps").
    `complete` is injected so the loop is testable offline."""
```

**Example I/O:**
```text
# complete returns a tool call, then a final answer:
run_agent(msgs, repo, scripted_complete, BudgetTracker(5, 10_000)).stop_reason  -> "answered"
# a dispatch that raises once, then the model adapts:
run_agent(...).stop_reason            -> "answered"   (recovered)
# a model that always repeats the same call:
run_agent(...).stop_reason            -> "stuck"
# a model that always calls a (different) tool and never answers:
run_agent(...).stop_reason            -> "budget" or "max_steps"
```

### `evaluate.py` — `learner`

**Purpose:** Score a run — the agent-evaluation core. (M4.)

**Key function (learner-owned core):**

```python
def evaluate_run(result: RunResult, expected: dict) -> dict:
    """expected = {"answer_contains": str, "optimal_steps": int}. Return:
       {"completed": stop_reason=="answered" AND answer_contains in (result.answer or ""),
        "steps": result.steps, "tool_calls": len(result.history),
        "efficiency": min(1.0, optimal_steps / max(result.steps, 1)),
        "stop_reason": result.stop_reason}."""
```

**Example I/O:**
```text
evaluate_run(answered_run(answer="...groq...", steps=3, 2 calls),
             {"answer_contains": "groq", "optimal_steps": 3})
   -> {"completed": True, "steps": 3, "tool_calls": 2, "efficiency": 1.0, "stop_reason": "answered"}
evaluate_run(stuck_run, {"answer_contains": "x", "optimal_steps": 2})
   -> {"completed": False, ..., "stop_reason": "stuck"}
```

### `agent_app.py` — `provided`

**Purpose:** The orchestrator. Builds a system prompt, a real LiteLLM-backed `complete`
(`tool_choice="auto"`), and a `BudgetTracker` from `config.py`, then runs `run_agent` on a multi-step
task against a repo, printing the ReAct trace, the final answer + `stop_reason`, and the
`evaluate_run` summary. Calls the learner modules, so it raises `NotImplementedError` until they're
done — then runs the agent end to end. `python agent_app.py "task" --repo .`

### `tests/` — `provided`

`test_safety.py` (M1 detect_stuck, M2 BudgetTracker), `test_agent.py` (M3 run_agent — a fake
`complete` scripts the model; a monkeypatched dispatch tests recovery), `test_evaluate.py`
(M4 evaluate_run), `test_config_models.py` (drift guard, passes today), `conftest.py` (puts `code/`
on `sys.path`, builds a throwaway fixture repo, and provides scripted-model helpers). All offline —
no provider, no network.

---

## Input / Output Contracts

| Function | Input | Expected Output | Error Behavior |
|----------|-------|-----------------|----------------|
| `detect_stuck(history, window)` | `list[Action]`, `int` | `bool` — True iff last `window` identical | `< window` items → `False` (never raises) |
| `BudgetTracker.tick(tokens)` | `int` | mutates `steps`/`tokens` | — |
| `BudgetTracker.over_budget()` | — | reason `str` if crossed, else `None` | — |
| `run_agent(messages, root, complete, budget, …)` | messages, root, fn, budget | `RunResult` with a `stop_reason` | a raising tool → recovered (fed back), never fatal |
| `evaluate_run(result, expected)` | `RunResult`, `dict` | `dict` (completed/steps/tool_calls/efficiency/stop_reason) | missing `answer` → treated as not-completed |
| `parse_tool_calls(message)` | OpenAI-style message | `list[ToolCall]` (args json.loads'd) | no `tool_calls` → `[]` (provided, carried) |

---

## Extended Requirements

Beyond the core implementation — complete these after the core is working.

- [ ] **Explicit planning:** a `plan_task` step that decomposes the task before acting; measure steps-to-answer with vs. without (source: `sources/articles/building-effective-agents.md`).
- [ ] **Reflection loop:** after answering, a self-check step that continues if the task isn't satisfied — evaluator-optimizer (sources: `sources/articles/building-effective-agents.md`, `sources/papers/generative-agents.md`).
- [ ] **Structured final output:** a forced `emit_answer(answer, steps_taken, confidence)` tool (source: `sources/official-docs/anthropic-tool-use.md`).
- [ ] **Real cost budget:** add `max_usd` to `BudgetTracker` using your provider's official per-MTok price (source: `sources/official-docs/anthropic-pricing.md`).
- [ ] **Retry with backoff:** retry a transient tool error once before feeding the error back.

---

## Known Difficulty Spikes

Listed so the learner expects them, not so they can avoid them.

1. **Stuck vs. slow.** `detect_stuck` must compare the *whole* action (tool + args). Comparing the tool name alone kills a productive agent reading many different files; comparing nothing waits for `max_steps`.
2. **Recovery is feed-back, not swallow.** Wrapping dispatch in `try/except` is half of it; the other half is appending the error *as an observation* so the model adapts. `except: pass` causes a stuck loop and is worse than crashing.
3. **Distinct stop reasons.** The single most useful output of the loop is *why* it stopped. Returning a bare string answer (or a bool) throws away the information evaluation depends on — thread a `stop_reason` through every return path.
4. **Token budget ≠ step budget.** A step cap won't catch a few enormous steps. Track tokens too, and check both every iteration.
5. **Message ordering (carried from P06).** Append the assistant turn before the tool-result messages, or the next provider call errors on an orphan tool result.
6. **Judgment is graded.** The "don't use an agent" ablation (M6) is a required experiment, not optional flavor — you must produce a task where the loop is measurably worse than a single call.

---

## Debugging Approach

When things break, check in this order:

1. Environment — is `.env` loaded? (Only `agent_app.py`'s live run needs a provider; all tests run offline.)
2. Safety — `python -m pytest tests/test_safety.py`: does `detect_stuck` (M1) flag identical-only, and does `BudgetTracker` (M2) return a reason when over?
3. Loop — `python -m pytest tests/test_agent.py`: does `run_agent` return `answered` on a scripted tool-then-answer, `answered` (recovered) when dispatch raises, and `stuck`/`budget` on the pathological models?
4. Recovery — temporarily make a tool raise in the live app: does the run survive and the model adapt, or does it crash? (If it crashes, your `try/except` is missing or too narrow.)
5. Evaluate — `python -m pytest tests/test_evaluate.py`: is `efficiency` `<= 1.0` and `completed` keyed off `stop_reason == "answered"` *and* the substring?
6. Live — `python agent_app.py "what model does this repo default to?" --repo .`: read the printed trace — does it act over several steps, then answer? Does the eval summary look right?
7. Source — re-read `source/lesson.agent.md` §3 and `sources/articles/building-effective-agents.md` (when *not* to use an agent).

---

## Integration Notes

**Depends on:** Project 06 (the tool-use loop — sandbox, schemas, dispatch, parsing — all carried in
*provided*; this project is that loop made reliable), Project 01 (token-cost arithmetic — the budget
is this enforced live), Project 07 (evaluation discipline — `evaluate_run` is that applied to a run).

**Depended on by:** Project 09 (Personal Learning OS — orchestrates memory, retrieval, this agent, and
evaluation into one system; the reliability layer here is what makes it safe for P09 to let an agent
act over personal data). The budget / stuck / recovery / evaluation machinery is the substrate every
autonomous capability in P09 (and in StarcallOS) inherits.