# Project 04: PDF Research Assistant

# source/project.md — Detailed Project Specification

> Generated by: skills/lesson-generator
> For the high-level overview see the root PROJECT.md and the teaching content in
> `source/lesson.agent.md`.
> This file contains implementation contracts, not teaching content.

---

## Definition of Done

The project is complete when all of the following are true:

- [ ] Core implementation runs without errors (chunk → index → retrieve → grounded answer → faithfulness)
- [ ] All project milestones M1–M6 demonstrated (see `lesson.agent.md` §7)
- [ ] Documents are split into **overlapping, id-tagged chunks** (not one vector per document)
- [ ] The generator answers **only from the retrieved context**, **cites** the chunk id(s) used, and
      **refuses** ("I don't know") when the answer is not in the context
- [ ] `faithfulness.py` reports a **faithfulness score** `F = |V|/|S|` over the answer's claims (a number)
- [ ] UNDERSTANDING.md completed before first line of code
- [ ] FAILURE_ANALYSIS.md contains ≥3 intentional experiments (e.g. off-doc refusal, forced hallucination, chunk-size effect)
- [ ] EVALUATION.md contains quantitative results (faithfulness, retrieval hit/miss), not impressions
- [ ] STARCALLOS_REFLECTION.md identifies at least one concrete applicable pattern
- [ ] Guiding tests in `code/tests/` pass (`python -m pytest`)

---

## File Specification

Eight modules in `code/`. Retrieval is **reused from Project 03** (provided complete) so the friction
here is the RAG-specific work: chunking, grounded generation, citation, and faithfulness.

### `config.py` — `provided`

Canonical provider-resolution block (do not edit) + a per-project `Config` adding `chunk_size`,
`chunk_overlap`, `top_k`, and a low default `temperature` (0.0 — grounded answering wants faithful
extraction, not creativity).

### `pdf_loader.py` — `provided`

**Purpose:** Parse a PDF to text via `pypdf`; ships `SAMPLE_DOC` so the pipeline + tests run with no
file and no network. PDF parsing is plumbing, not the learning target.

**Key functions:** `load_pdf(path: str) -> str`, constant `SAMPLE_DOC`.

### `embedding_helpers.py` — `provided`

**Purpose:** Wrap `litellm` embeddings (same as Project 03). `embed_many` / `embed_one`. The model is
chosen by `config.py`; pin ONE model for chunks and query.

### `retriever.py` — `provided` (Project 03 reuse)

**Purpose:** The "R" in RAG. Build a Chroma collection from chunks and retrieve top-k. Provided complete
because retrieval was Project 03's learning target.

**Key functions:** `build_index(chunks: list[Chunk]) -> Collection`, `retrieve(collection, query, k) -> list[Chunk]`.

### `chunker.py` — `learner`

**Purpose:** Split document text into overlapping, id-tagged chunks. (Milestone M1.)

**Key function (learner-owned core):**

```python
def chunk_text(text: str, chunk_size: int, overlap: int, source: str = "doc") -> list[Chunk]:
    """Fixed-size windows over whitespace tokens, stepping by (chunk_size - overlap). Each chunk gets
    a stable id ('c0', 'c1', ...), its source, and start index. Stop after the window that reaches the
    end (no redundant tail). Empty text -> []."""
```

**Provided:** the `Chunk` dataclass (`id`, `text`, `source`, `start`), `main()`.

**Does not:** embed or retrieve (that's `retriever.py`).

**Example I/O:**
```text
chunk_text("A B C D E F G H I J", chunk_size=4, overlap=2)
→ [Chunk("c0","A B C D"), Chunk("c1","C D E F"), Chunk("c2","E F G H"), Chunk("c3","G H I J")]
# step = 2; adjacent chunks share 2 tokens; 4 chunks, no redundant tail
```

### `generator.py` — `learner`

**Purpose:** The core — ground the LLM in the retrieved chunks and cite them. (Milestones M3, M4.)

**Key functions (learner-owned core):**

```python
def build_prompt(query: str, chunks: list[Chunk]) -> list[dict]:
    """Return [system, user] messages. System: answer ONLY from context, cite chunk ids like [c0],
    and say you don't know if the answer is absent (refusal). User: a context block of '[id] text'
    lines, then the question."""

def answer(query: str, chunks: list[Chunk]) -> Answer:
    """Call litellm.completion over build_prompt(...) at cfg.temperature; return Answer(text, citations)
    where citations are the chunk ids found in the text (via the provided _CITE regex)."""
```

**Provided:** `Answer` dataclass, `_CITE` regex, `main()`.

**Does not:** retrieve or score faithfulness.

**Example I/O:**
```text
build_prompt("What was Q3 revenue?", [Chunk("c0","Q3 revenue was 4.2M, up 8% YoY."), Chunk("c1","...")])
→ [{"role":"system", "content": "...answer only from context...say you don't know..."},
   {"role":"user",   "content": "[c0] Q3 revenue was 4.2M...\n[c1] ...\n\nQuestion: What was Q3 revenue?"}]

answer("What was Q3 revenue?", [Chunk("c0","Q3 revenue was 4.2M, up 8% YoY.")])   # needs a provider
→ Answer(text="Q3 revenue was 4.2M [c0].", citations=["c0"])
# off-document question → text says it doesn't know, citations == []
```

### `faithfulness.py` — `learner`

**Purpose:** Detect hallucination by measuring faithfulness. (Milestone M5.)

**Key functions (learner-owned core):**

```python
def faithfulness_score(verdicts: list[bool]) -> float:
    """F = |V|/|S| (supported claims / total). Empty list -> 1.0 (vacuously faithful)."""

def check_faithfulness(answer_text: str, chunks: list[Chunk], claims: list[str] | None = None) -> FaithfulnessReport:
    """Split answer into claims, verify each against the joined context via an LLM yes/no call, score it."""
```

**Provided:** `FaithfulnessReport` dataclass, `_VERIFY_SYSTEM` prompt, `main()` (pure score demo).

**Example I/O:**
```text
faithfulness_score([True, True, False])  →  0.6667
faithfulness_score([])                   →  1.0
check_faithfulness("Q3 revenue was 4.2M. Margin was 40%.", [Chunk("c0","Q3 revenue 4.2M, margin 21%.")])
→ FaithfulnessReport(score=0.5, claims=[...2...], verdicts=[True, False])   # needs a provider
```

### `rag.py` — `provided`

**Purpose:** The end-to-end orchestrator wiring ingest → chunk → index → retrieve → answer → verify.
Calls the learner modules, so it raises `NotImplementedError` until they're done — then prints the
answer, its citations, and the faithfulness score.

---

## Input / Output Contracts

| Function | Input | Expected Output | Error Behavior |
|----------|-------|-----------------|----------------|
| `chunk_text(text, size, overlap)` | `str`, `int`, `int` | `list[Chunk]`, adjacent overlap by `overlap`, stable ids | empty text → `[]`; `overlap >= size` → raise/guard |
| `build_prompt(query, chunks)` | `str`, `list[Chunk]` | `[system, user]` dicts; ids + text + refusal instruction present | empty chunks → prompt still valid (model should refuse) |
| `answer(query, chunks)` | `str`, `list[Chunk]` | `Answer(text, citations)`; citations ⊆ chunk ids | off-doc question → refusal text, `citations == []` |
| `faithfulness_score(verdicts)` | `list[bool]` | `float` in `[0,1]` = mean of bools | `[]` → `1.0` (documented) |
| `check_faithfulness(text, chunks)` | `str`, `list[Chunk]` | `FaithfulnessReport(score, claims, verdicts)` | no claims → score `1.0` |

---

## Extended Requirements

Beyond the core implementation — complete these after the core is working.

- [ ] **Char/page citations:** carry each chunk's `start`/page so a citation points to an exact
      location, mirroring `char_location`/`page_location` (source: `sources/official-docs/anthropic-citations.md`).
- [ ] **Compare two chunking strategies** (fixed-size vs recursive/sentence) on the same questions;
      report the difference in retrieval correctness and faithfulness.
- [ ] **Context relevance** (`CR = |extracted|/|total sentences|`) to tell whether a weak answer is a
      retrieval/chunking problem or a generation problem (source: `sources/papers/ragas.md`).
- [ ] Persist the index with a Chroma `PersistentClient` (Project 03) so re-runs skip re-embedding.

---

## Known Difficulty Spikes

Listed so the learner expects them, not so they can avoid them.

1. **Grounding that actually grounds.** A weak system prompt lets the model answer from parametric
   memory and ignore the context — and the answer often *looks* right. Test it with an off-document
   question: a grounded system refuses; an ungrounded one invents an answer.
2. **Chunk overlap off-by-one.** `step = chunk_size - overlap`; getting the window stride wrong either
   drops boundary text or emits a redundant tail chunk. The guiding test pins the exact 4-chunk output.
3. **Citation parsing vs. citation *behavior*.** The `_CITE` regex extracts `[c0]` from the text, but
   the model only emits `[c0]` if your prompt told it to and the chunk ids are visible — the prompt is
   the hard part, not the regex.
4. **Faithfulness needs claim decomposition.** Scoring a whole answer as one true/false misses partial
   hallucination. Split into sentence-level claims and verify each.
5. **Same embedding model both sides** (carries from Project 03) — mixing models makes retrieval return
   garbage and the whole pipeline fails silently.

---

## Debugging Approach

When things break, check in this order:

1. Environment — is `.env` loaded? Is a provider available (`USE_OLLAMA=1` or a key)?
2. Chunking — `python chunker.py`: do chunks overlap and carry ids? (offline)
3. Retrieval — does the question retrieve the chunk that actually contains the answer? Print the hits.
4. Grounding — print the messages from `build_prompt`: are the chunk ids + text in the prompt, and is
   the refusal instruction present?
5. Memory vs. context — ask an **off-document** question. If it answers confidently, grounding is
   broken (the model is using parametric memory).
6. Faithfulness — does `faithfulness_score` pass its offline test? If yes, a bad score is a verifier or
   claim-splitting issue, not the math.
7. Source — re-read `source/lesson.agent.md` §3 and `sources/papers/rag-paper.md` / `ragas.md`.

---

## Integration Notes

**Depends on:** Project 03 (retrieval — `build_index`/`retrieve` reused; persistent cosine index;
same-model-both-sides rule), Project 01 (chat completion via LiteLLM; system + user messages).

**Depended on by:** Project 05 (memory — RAG over your own past), Project 07 (evaluation framework —
generalizes the faithfulness pass into a full LLM-as-judge suite). The chunk → retrieve → ground →
verify skeleton recurs throughout the curriculum.