# Project 07: AI Evaluation Framework

# source/project.md — Detailed Project Specification

> Generated by: skills/lesson-generator
> For the high-level overview see the root PROJECT.md and the teaching content in
> `source/lesson.agent.md`.
> This file contains implementation contracts, not teaching content.

---

## Definition of Done

The project is complete when all of the following are true:

- [ ] Core implementation runs without errors (dataset → judge → parse → summarize → regression report)
- [ ] All project milestones M1–M6 demonstrated (see `lesson.agent.md` §7)
- [ ] The judge prompt uses a fixed scale, asks for **reasoning before the score**, and is **reference-guided** when a reference is provided
- [ ] `parse_judge_score` handles JSON and plain text, **clamps** out-of-range scores, and **raises** (never returns 0) on a no-score reply
- [ ] `summarize` reports a mean **and** a pass-rate against a threshold
- [ ] `compare_runs` flags **per-case** regressions/improvements over the **common** cases, not just a mean delta
- [ ] UNDERSTANDING.md completed before first line of code
- [ ] FAILURE_ANALYSIS.md reproduces ≥1 judge bias (verbosity/position/self-enhancement) **and** a regression hidden in a flat mean — quantified
- [ ] EVALUATION.md contains quantitative results (means, pass-rates, deltas, a judge-vs-human spot-check), not impressions
- [ ] STARCALLOS_REFLECTION.md identifies at least one concrete applicable pattern
- [ ] Guiding tests in `code/tests/` pass (`python -m pytest`)

---

## File Specification

Six modules in `code/` (plus tests). The model call, dataset, and reporting are **provided**, so the
friction is the eval-specific work: the **judge prompt + parsing**, **aggregation**, and **regression**.

### `config.py` — `provided`

Canonical provider block (do not edit) + a per-project `Config` adding `judge_model` (the grader —
ideally *not* the model under test), `judge_scale` (5), `pass_threshold` (4), `regression_tolerance` (0).

### `dataset.py` — `provided`

`TestCase` (`id`, `question`, optional `reference`), a built-in `load_sample_cases()`, and a
`load_jsonl()` loader. The dataset must be **frozen** between baseline and candidate runs.

### `llm_judge.py` — `partial`

**Purpose:** LLM-as-a-judge — prompt + parse. (M1, M2.)

**Provided:** the `JudgeResult` dataclass (`case_id`, `score`, `reasoning`); `judge(...)` — the
`litellm` call (temp 0) wiring the two learner functions together.

**Key functions (learner-owned core):**

```python
def build_judge_prompt(question, answer, *, reference=None, scale=5) -> list[dict]:
    """System+user messages: fixed 1..scale scale, REASONING BEFORE SCORE, parseable format,
    reference-guided when reference is given."""

def parse_judge_score(text, *, scale=5) -> tuple[int, str]:
    """Extract an int score in [1, scale] from the judge's prose (JSON / 'Score: N' / 'N/scale').
    Clamp out-of-range; RAISE ValueError on no-score (never silently 0). Return (score, reasoning)."""
```

**Example I/O:**
```text
build_judge_prompt("Q","A", reference="GOLD", scale=5) -> [system_msg(...scale...reason...score...), user_msg(...Q...A...GOLD...)]
parse_judge_score('{"reasoning":"good","score":4}')    -> (4, "good")
parse_judge_score("I'd rate this 3/5.")                 -> (3, "...")
parse_judge_score("Score: 9", scale=5)                  -> (5, ...)     # clamped
parse_judge_score("no number")                          -> ValueError
```

### `metrics.py` — `partial`

**Purpose:** Aggregate scores into a decision. (M3.)

**Provided:** `Summary` dataclass; `exact_match` / `contains` (cheap reference-based metrics).

**Key function (learner core):**
```python
def summarize(results: list[JudgeResult], *, pass_threshold=4) -> Summary:
    """Summary(n, mean_score, pass_rate). pass_rate = fraction with score >= threshold. Guard n==0."""
```
**Example I/O:** `summarize([5,4,3,2]@4) -> Summary(4, 3.5, 0.5)`; `summarize([]) -> Summary(0, 0.0, 0.0)`.

### `regression_runner.py` — `partial`

**Purpose:** Detect regressions between two runs. (M4.)

**Provided:** `RegressionReport` dataclass; `run_eval(cases, answer_fn)` (dataset → judge → results);
`main()` CLI demo.

**Key function (learner core):**
```python
def compare_runs(baseline, candidate, *, tolerance=0) -> RegressionReport:
    """Match by case_id (COMMON cases only). delta = cand - base; regression if delta < -tolerance,
    improvement if delta > tolerance. Means over common cases. Surfaces a drop a flat mean hides."""
```
**Example I/O:** base `{a4,b5,c3}` vs cand `{a4,b2,c5}`, tol 0 → regressions `["b"]`, improvements `["c"]`, means 4.0→3.67.

### `report.py` — `provided`

`format_summary(Summary)` and `format_regression(RegressionReport)` — readable text for a PR/console.

### `tests/` — `provided`

`test_llm_judge.py` (M1, M2), `test_metrics.py` (M3), `test_regression.py` (M4),
`test_config_models.py` (drift guard, passes today), `conftest.py` (puts `code/` on `sys.path`). All
offline — parsing/aggregation/regression are pure; the live judge call is not tested.

---

## Input / Output Contracts

| Function | Input | Expected Output | Error Behavior |
|----------|-------|-----------------|----------------|
| `build_judge_prompt(q, a, reference, scale)` | strings, opt ref, int | `list[{role,content}]` (system+user) with scale, reasoning-first, ref when given | — |
| `parse_judge_score(text, scale)` | `str`, int | `(score:int in [1,scale], reasoning:str)` | no score → `ValueError`; out-of-range → clamp |
| `summarize(results, pass_threshold)` | `list[JudgeResult]`, int | `Summary(n, mean_score, pass_rate)` | empty → `Summary(0, 0.0, 0.0)` |
| `compare_runs(base, cand, tolerance)` | two `list[JudgeResult]`, int | `RegressionReport(regressions, improvements, means, delta)` | no common cases → empty lists, means 0.0 |
| `judge(case_id, q, a, reference)` *(provided)* | strings | `JudgeResult` | depends on M1/M2 + a provider |

---

## Extended Requirements

Beyond the core implementation — complete these after the core is working.

- [ ] **Pairwise judging + position-swap:** compare two answers, run both orders, and require/average agreement to cancel position bias (source: `sources/papers/mt-bench.md`).
- [ ] **Reference-guided lift:** measure judge–human agreement with and without a reference, and report the difference (source: `sources/papers/mt-bench.md`).
- [ ] **Faithfulness metric:** decompose an answer into claims and verify each against a context — a reference-free judge (`F = |V|/|S|`), reusing Project 04 (source: `sources/papers/ragas.md`).
- [ ] **Cost/latency tracking** per eval run (reuse Project 01's cost math).
- [ ] **Verbosity audit:** log answer length alongside score and report the correlation (source: `sources/papers/mt-bench.md`).

---

## Known Difficulty Spikes

Listed so the learner expects them, not so they can avoid them.

1. **Robust parsing is the load-bearing step.** The judge replies in prose and the format drifts. Handle JSON *and* text, clamp out-of-range, and raise on no-score — a silent `0` poisons every aggregate downstream.
2. **Reasoning must come before the score.** A score emitted before its justification is a worse, less consistent score; put the reasoning field first in the requested format (source: `sources/papers/mt-bench.md`).
3. **A flat mean hides per-case regressions.** `compare_runs` exists precisely because a 5→2 drop and a 3→5 rise cancel in the average; compare like-for-like by `case_id`.
4. **Freeze the dataset.** If cases differ between baseline and candidate, the comparison is meaningless. Version the dataset.
5. **Judge ≠ subject.** Judging a model with itself inflates scores (self-enhancement bias); default `judge_model` to a different/stronger model (source: `sources/papers/mt-bench.md`).
6. **Verbosity bias is a metric trap.** If "better" correlates with "longer," a change that only lengthens answers looks like a win. Watch length (source: `sources/papers/mt-bench.md`).

---

## Debugging Approach

When things break, check in this order:

1. Environment — only the live `judge`/`run_eval` calls need a provider; all tests run offline.
2. Judge prompt — `python -m pytest tests/test_llm_judge.py -k build`: does the prompt include the answer, the scale, the reference, and ask for reasoning before score?
3. Parsing — `python -m pytest tests/test_llm_judge.py -k parse`: JSON, "Score: N", "N/scale", clamp, and raise-on-garbage all handled?
4. Aggregate — `python -m pytest tests/test_metrics.py`: mean and pass-rate right, empty guarded?
5. Regression — `python -m pytest tests/test_regression.py`: per-case flags correct, only common cases, tolerance honored?
6. Live — `python regression_runner.py`: does the demo judge the sample set and print a summary? Print a couple of `JudgeResult.reasoning` values — do the scores look sane?
7. Source — re-read `source/lesson.agent.md` §3 and `sources/papers/mt-bench.md` (biases + mitigations).

---

## Integration Notes

**Depends on:** Project 01 (the judge is one `litellm.completion`; `temperature=0`), Project 04
(faithfulness/RAGAS — the worked metric this generalizes; reference-guided judging), Project 02
(embeddings for the answer-relevance extension). Any prior project is a *system under test*.

**Depended on by:** Project 08 (agent — "did the agent succeed?" is a judge call over a transcript)
and Project 09 (learning OS — measuring whether the system helps). The regression harness is what lets
every later project change prompts/models without flying blind; Project 05's LLM-rated importance was a
special case of the LLM-as-a-judge this project generalizes.