AI Evaluation Framework

Learning Objectives

By the end of this project, you will be able to:

• Explain why "it seemed to work" is not an evaluation, and why text outputs with no single right answer can't be checked with assert ==.
• Implement LLM-as-a-judge: build a prompt that scores an answer on a scale, and justify why a strong judge is usable (~80% human agreement).
• Name and reproduce the biases of an LLM judge — position, verbosity, self-enhancement, weak math grading — and say which one caused a bad score.
• Apply the standard mitigations: reasoning-before-score (chain-of-thought), reference-guided judging, and position-swapping.
• Parse a judge's free-text reply robustly into a numeric score (JSON, "Score: 4", "4/5"), clamping out-of-range values.
• Aggregate per-case scores into a decision: a mean score and a pass-rate against a threshold.
• Build a regression test: freeze a dataset, score a baseline, re-score after a change, and flag the cases that got worse.
• Distinguish reference-free metrics (faithfulness/relevance) from reference-based and judge-based scoring.

1. Motivation

Why This Exists

You changed your system prompt. Is the assistant better or worse? With code you'd run the tests — but there is no assert summary == expected, because a good summary, a good answer, a good explanation has no single correct string. "Most AI developers ship based on vibes and demo impressions. Production failures happen because no one built evals." The breakthrough is LLM-as-a-judge: a strong model can grade open-ended answers and "match both controlled and crowdsourced human preferences well, achieving over 80% agreement — the same level of agreement between humans." That makes quality measurable, and measurable means improvable.

What Breaks Without It

Without an eval harness, every change is a gamble — you ship and hope, and regressions slip through because no one was watching a number. And if you build a judge but ignore its biases, your metric lies to you: a prompt change that only makes answers longer looks like an improvement (verbosity bias).

Real-World Stakes

Every serious AI product has an eval harness behind it; the ones that don't ship regressions they can't see. "Evaluation infrastructure is the invisible moat… Braintrust, LangSmith, and others are built on selling evaluation infrastructure." The judgment that separates a pro from an amateur: do you trust a single demo, or do you have a dataset, a judge you've validated, and a pass-rate you watch on every change?

2. Mental Model

Explain Like I'm 12

Imagine grading 500 essays. There's no answer key — a good essay can be written a thousand ways — so you hire a really sharp teaching assistant to read each one and give it a 1–5. That TA (the judge) is fast and mostly agrees with you. But the TA has quirks: they give higher marks to longer essays, and to whichever essay they read first. So you make two rules: "write down your reasons before you give the number," and "if you're comparing two, read them in both orders." Now their grades are trustworthy enough that when you change how you teach, you can re-grade the same 500 essays and see whether scores went up or down. That up-or-down check is a regression test; the TA-with-rules is LLM-as-a-judge.

Explain Like I'm a Software Engineer

• A judge is an LLM call whose output is a score, not an answer. build_judge_prompt(question, answer, reference?) → litellm.completion → free text → parse_judge_score → an int in [1, scale].
• The judge prompt encodes the mitigations: a fixed scale (1–5), reasoning before the score (chain-of-thought — improves agreement and is auditable), and the reference when you have one (reference-guided).
• The model returns prose, so you must parse robustly: JSON {"score": 4}, "Score: 4", or "I'd give this 4/5." Extract the integer, clamp to range, fail loud only when truly unparseable.
• A run produces list[JudgeResult]. summarize → (mean, pass_rate@threshold). compare_runs(baseline, candidate) → which case_ids dropped beyond a tolerance = regressions.

Real-World Analogy

How It Works (Diagram)

 ONE CASE:
   question: "Summarize the refund policy."
   answer:   "<the system-under-test's output>"
   reference?: "<gold summary, if you have one>"
        │
        ▼  build_judge_prompt  (scale 1–5, "reason FIRST, then score", include reference)
   judge LLM ──► "The answer covers X and Y but misses the 30-day window. Score: 3"
        │
        ▼  parse_judge_score  → 3   (robust to JSON / '3/5' / 'Score: 3'; clamp to [1,5])
   JudgeResult(case_id, score=3, reasoning="...misses the 30-day window")

 MANY CASES:
   summarize([...]) → mean=3.8, pass_rate@4 = 0.62
   compare_runs(baseline, candidate, tolerance=0) → regressions=[case_7, case_12]

3. Technical Explanation

Formal Definition

An LLM judge is a function (question, answer[, reference]) → score ∈ {1..N} realized by an LLM call. Single-answer grading scores one answer; pairwise compares two. Agreement is how often the judge matches a human; GPT-4 reaches ">80% agreement, the same level of agreement between humans." Pass-rate = |{c : score(c) ≥ t}| / |C|; a regression (tolerance τ) is candidate.score < baseline.score − τ.

How It Works Step by Step

1. Design the dataset. A case is (id, question[, reference]). Good cases are representative, include edge inputs, and are frozen — the dataset must not change between baseline and candidate.
2. Build the judge prompt. A system message with the role and scale; a user message with the question, the answer, and the reference if present. Bake in the mitigations: reasoning, then the score, in a parseable format.
3. Call the judge. One litellm.completion at temperature=0 for repeatability. The judge can (and often should) differ from the model under test — avoid self-enhancement bias.
4. Parse the score. The reply is prose. Try JSON; else regex "score: N" or "N/scale"; else first in-range integer. Clamp to [1, scale]. Raise only when there is no number at all — silent 0s poison your mean.
5. Aggregate. summarize → mean + pass-rate. One number you trust beats 200 you scroll past.
6. Regression-test. Score the baseline once and store it; after a change, score the candidate on the same dataset and compare_runs per case_id.

Key Concepts

4. Guided Examples

The lab stack: litellm (the judge call via config.py), pure-Python parsing/aggregation, and a small frozen dataset. Examples mirror the guiding tests.

Example 1: Simplest Case — parse a judge's reply into a score

from llm_judge import parse_judge_score

parse_judge_score('{"reasoning": "covers it well", "score": 4}', scale=5)  # (4, "covers it well")
parse_judge_score("Reasoning: solid but terse.\nScore: 5", scale=5)        # (5, "Reasoning: solid but terse.")
parse_judge_score("I'd rate this 3/5.", scale=5)                            # (3, "I'd rate this 3/5.")
parse_judge_score("Score: 9", scale=5)                                      # (5, ...)  ← clamped to scale

Example 2: Real-World Case — aggregate scores into a decision

from llm_judge import JudgeResult
from metrics import summarize

results = [
    JudgeResult("c1", 5, "..."), JudgeResult("c2", 4, "..."),
    JudgeResult("c3", 3, "..."), JudgeResult("c4", 2, "..."),
]
s = summarize(results, pass_threshold=4)
print(s.n, s.mean_score, s.pass_rate)   # 4  3.5  0.5   (two of four scored >= 4)

Example 3: When It Fails — a regression hides inside a flat average

from llm_judge import JudgeResult
from regression_runner import compare_runs

baseline  = [JudgeResult("a", 4, ""), JudgeResult("b", 5, ""), JudgeResult("c", 3, "")]
candidate = [JudgeResult("a", 4, ""), JudgeResult("b", 2, ""), JudgeResult("c", 5, "")]

report = compare_runs(baseline, candidate, tolerance=0)
print(report.regressions)    # ['b']   — dropped 5 → 2
print(report.improvements)   # ['c']   — rose 3 → 5
print(round(report.baseline_mean, 2), round(report.candidate_mean, 2))  # 4.0 3.67

5. Reflection Before Building

After filling in UNDERSTANDING.md, use the AI mentor pattern in docs/meta/learning-flow.md to get feedback. Record the feedback in UNDERSTANDING_FEEDBACK.md.

6. Project Assignment

See PROJECT.md and source/project.md for the full specification including file structure and extended requirements.

Core Requirement

Build a reusable evaluation harness in code/:

• llm_judge.py — build_judge_prompt (bias-mitigated prompt) and parse_judge_score (robust extraction) are the learner core; judge and JudgeResult are provided.
• metrics.py — summarize (mean + pass-rate) is the learner core; exact_match/contains are provided.
• regression_runner.py — compare_runs (baseline vs candidate → regressions) is the learner core; the run-a-dataset harness is provided.
• dataset.py / report.py — frozen test cases + formatting. (Provided.)

Extended: pairwise judging with position-swap; reference-guided agreement lift; a faithfulness metric (reuse Project 04); cost/latency tracking; a verbosity audit.

7. Project Milestones

Work through these in order. Each milestone should produce runnable code before moving on.

8. Self-Evaluation

After building, honestly evaluate your implementation against these criteria. Record your answers in EVALUATION.md.

9. Common Mistakes

10. Connections

Builds On

The judge is Project 01's litellm.completion — same call, but its job is to output a score. The faithfulness metric you build as an extension is Project 04's RAGAS work generalized (F = |V|/|S|), and reference-guided judging is the same instinct as grounding an answer in retrieved context (Projects 03/04). Any prior project is a system under test you can now measure instead of eyeball.

Enables

Project 08 (agent) needs evaluation badly — agents fail in long multi-step trajectories, and "did the agent succeed?" is itself a judge call over a transcript. Project 09 (personal learning OS) uses evaluation to know whether the system is actually helping. The LLM-rated importance from Project 05 was a special case of LLM-as-a-judge; this project is the general tool.

Production Patterns

Real eval stacks (Braintrust, LangSmith, OpenAI Evals) are this plus scale: versioned datasets, LLM-as-judge with validated prompts, pairwise + position-swap, reference-guided scoring, per-case dashboards, regression gates in CI, and cost/latency tracking per run. The frontier adds human-in-the-loop spot-checks to calibrate the judge and catch the ~20% it gets wrong.

StarcallOS Relevance

Sources

See source/resources.md for the complete annotated source list.