PDF Research Assistant
Ground your AI in real documents with RAG
Learning Objectives
By the end of this project, you will be able to:
- Explain RAG as combining parametric memory (the LLM's weights) with non-parametric memory (a retrieved, updatable document store), and why that beats a bigger model for document QA.
- Implement the full pipeline: ingest → chunk → embed → index → retrieve → generate → cite → verify.
- Design a chunking strategy and reason about the size tradeoff (small = precise but context-poor; large = rich but imprecise), including overlap.
- Construct a grounded-generation prompt that answers only from the retrieved context and refuses ("I don't know") when the answer is absent.
- Build a citation system that traces each claim back to the source chunk it came from.
- Detect hallucination by measuring faithfulness:
F = |supported| / |total|over the answer's claims. - Name and reproduce the major failure modes of RAG and tell which stage caused each.
1. Motivation
Why This Exists
An LLM "stores factual knowledge in its parameters," but its ability to access that knowledge precisely is limited, and "providing provenance for [its] decisions and updating [its] world knowledge remain open research problems." Translation: ask a raw LLM about your PDF — a contract, a paper, last quarter's report — and it cannot. The document isn't in its weights, it has a training cutoff, and even when it guesses right it can't tell you where it got the answer. RAG fixes all three by giving the model a retrieved, non-parametric memory at query time.
You have documents the model has never seen, and answers that must be grounded (from the document, not invented) and attributable (you can check the source). Stuffing the whole document into the prompt doesn't scale and buries the answer in noise. RAG: chunk the document, retrieve only the relevant passages, and answer from those with citations.
What Breaks Without It
The danger is silent. A RAG system that retrieves the wrong chunk, or answers from the model's memory instead of the chunk, produces a fluent, confident, wrong answer with no error. In a contract or a clinical setting that isn't a bug ticket — it's liability.
Real-World Stakes
Every enterprise AI product either is RAG or contains RAG — legal research, medical documentation, financial analysis, support automation. The judgment that separates a demo from a shipped product: did retrieval find the right text, did the model stay grounded in it, and can you prove the answer came from the source.
This is the most commercially deployed AI architecture. Users pay for trustworthy answers over their documents; faithfulness and citations are what they're actually buying.
2. Mental Model
Explain Like I'm 12
Imagine an open-book test. A student who memorized the textbook (a plain LLM) will confidently make things up about a book they never read. Now give them your book and one rule: "Find the exact pages about the question, read only those, write your answer, and underline the sentence you got it from. If the book doesn't say, write 'not in the book.'" That's RAG. Chunking is tearing the book into index cards so they can find the right card fast. Citations are the underlines. Faithfulness is the teacher checking every sentence is backed by an underlined card — and catching the ones that aren't.
Explain Like I'm a Software Engineer
- RAG = retrieve-then-generate. Embed the question, retrieve the top-k nearest chunks from a vector DB (Project 03), concatenate them into the prompt as context, and ask the LLM to answer only from that context. It combines parametric memory (the weights) with non-parametric memory (your indexed chunks).
- Chunking is the upstream gate. You can't embed a 40-page PDF as one vector (too big for the model's context window; too diffuse to match a question), so you split it. Chunk size trades precision against context.
- Grounding is a prompt discipline: provide the chunks, instruct "answer only from the context; if it's not there, say you don't know," and tag each chunk with an id so the model can cite it.
- Faithfulness is a second pass: extract the answer's claims, check each against the context, score
F = |V|/|S|. The automated hallucination detector.
Real-World Analogy
RAG is a paralegal with a filing cabinet, not a know-it-all. They don't answer from memory — they pull the relevant files (retrieval), read only those, draft an answer, and staple a photocopy of the exact paragraph to each claim (citations). A good paralegal writes "the file doesn't address this" instead of guessing (refusal). Faithfulness is the partner who checks every claim against the stapled photocopies.
How It Works (Diagram)
INGEST (once):
report.pdf ──parse──► raw text ──CHUNK(size, overlap)──► [c0 c1 c2 ... cN]
│ embed each
▼
Chroma index (cosine)
ASK (per question):
"What was Q3 revenue?"
│ embed
▼
q-vector ──retrieve top-k──► [c7, c2, c9] (chunks most likely to hold the answer)
│ stuff into prompt as CONTEXT + ids
▼
LLM: "answer ONLY from context; cite ids; else say I don't know"
│
▼
"Q3 revenue was $4.2M [c7]." ──verify──► claim supported by c7? yes → faithful
3. Technical Explanation
Formal Definition
RAG: given a question q and a document store, retrieve a set of passages Z = top-k(q) and generate the answer from p(answer | q, Z). The store is non-parametric memory — a dense vector index accessed by a neural retriever — coupled to the parametric generator.
Faithfulness (RAGAS): extract the set of statements S the answer makes; let V ⊆ S be those that can be inferred from the context. Then F = |V| / |S| ∈ [0, 1] — the fraction of claims the retrieved context supports.
How It Works Step by Step
- Ingest & chunk. Parse the PDF to text, split into roughly fixed-size chunks with overlap ("fixed-sized chunking will be the best path in most cases"). Each chunk keeps an id and its source so it can be cited.
- Embed & index. Embed each chunk once into a Chroma collection (
space="cosine", Project 03). Built once; queried many. - Retrieve. Embed the question,
collection.query(...), get the top-k chunks with ids. The entire "R" of RAG. - Generate (grounded). Build a prompt: a system instruction ("answer only from context; cite the chunk ids; if absent, say you don't know"), the retrieved chunks labeled with ids, then the question. Call the LLM.
- Cite. A citation is a pointer from a span of the answer to the chunk id (and its page/char range) that supports the claim.
- Verify (faithfulness). Decompose the answer into claims; verify each against the context;
F = |V|/|S|. A low score flags hallucination.
Key Concepts
| Concept | Definition | Why It Matters |
|---|---|---|
| RAG | Retrieval-Augmented Generation: condition the answer on passages fetched at query time | Combines parametric + non-parametric memory; makes knowledge updatable and attributable |
| Parametric vs non-parametric memory | Parametric = facts in the weights; non-parametric = a swappable document index | You update non-parametric memory by re-indexing, not retraining — and you can cite it |
| Chunking | Splitting a document into smaller passages before embedding | A whole-PDF vector is too diffuse to match a question; chunk size sets the precision/context tradeoff |
| Chunk overlap | Repeating a slice of text across adjacent chunks | Stops a fact split across a boundary from being lost by both chunks |
| Grounding | Putting retrieved chunks in the prompt and answering only from them | Turns "the model's opinion" into "what the document says" |
| Refusal | Answering "I don't know" when the context lacks the answer | A RAG system that never refuses will confidently hallucinate off-document questions |
| Citation / provenance | A pointer from a span of the answer to the source chunk that supports it | Lets a human verify the claim — "track and verify information sources in responses" |
| Faithfulness | Fraction of the answer's claims the context supports: F = |V|/|S| | The hallucination metric — an unsupported claim is, by definition, made up |
A fluent answer is not a grounded answer. The hardest RAG failure is a confident answer drawn from the model's own memory while ignoring the retrieved context — it often looks right. That's why you measure faithfulness instead of eyeballing.
4. Guided Examples
The lab stack: pypdf (parse), chromadb (retrieve, Project 03), litellm (embeddings + chat via config.py). Examples mirror the guiding tests.
Example 1: Simplest Case — chunk text with overlap
from chunker import chunk_text
text = "A B C D E F G H I J" # 10 "tokens" (words) for the demo
chunks = chunk_text(text, chunk_size=4, overlap=2)
for c in chunks:
print(c.id, repr(c.text))
# c0 'A B C D'
# c1 'C D E F' # overlaps c0 by 2 tokens — nothing falls in a crack
# c2 'E F G H'
# c3 'G H I J'
Consecutive chunks share overlap tokens, so a fact straddling a boundary survives intact in a chunk. Each chunk has a stable id — that id is what you'll cite and what faithfulness checks against.
Example 2: Real-World Case — a grounded prompt that cites and can refuse
from chunker import Chunk
from generator import build_prompt
chunks = [
Chunk("c0", "Q3 revenue was $4.2M, up 8% YoY."),
Chunk("c1", "The company opened a Berlin office in July."),
]
messages = build_prompt("What was Q3 revenue?", chunks)
print(messages[0]["role"]) # 'system' → grounding + refusal instruction
print("[c0]" in messages[-1]["content"]) # True: chunk ids are in the prompt to cite
# A grounded model answers: "Q3 revenue was $4.2M [c0]." — note the citation to c0, not c1.
The retrieved chunks and their ids are in the prompt, and the system message both demands citations and permits refusal. The model is told to answer from the context, not from its own memory.
Example 3: When It Fails — faithfulness catches a hallucination
from faithfulness import faithfulness_score
# Claims from an answer, each verified against the retrieved context:
verdicts = [True, True, False] # 3rd claim ("revenue was $9M") is NOT supported by any chunk
print(faithfulness_score(verdicts)) # 0.6667 → 2 of 3 claims grounded
print(faithfulness_score([True, True, True])) # 1.0 — fully grounded
print(faithfulness_score([False, False])) # 0.0 — pure hallucination
F = |V|/|S| turns "did it hallucinate?" into a number. The third claim was fluent and confident — and unsupported. Without the faithfulness pass you'd ship it. This is the metric, not a feeling.
5. Reflection Before Building
Before writing any code, fill in UNDERSTANDING.md in your own words. No copy-pasting from the lesson. No AI assistance for writing — only for checking. If you can't explain it simply, go back and re-read.
Knowledge Check — Answer These in UNDERSTANDING.md
- In your own words, what does RAG add over (a) a raw LLM and (b) just stuffing the whole PDF into the prompt? Use "parametric vs non-parametric memory."
- Draw the ingest-once vs ask-per-question data flow. Which steps happen once, which per question?
- Chunk size: predict what goes wrong if chunks are too small, and what goes wrong if they're too large. Why does overlap help?
- Grounding can fail silently — the model answers from its own memory and the answer still looks right. How would you detect that this happened?
- What is faithfulness, and why is
F = |V|/|S|a hallucination detector? What's the difference between a faithful answer and a relevant one? - A RAG system is asked something the document doesn't cover. What should happen, and what makes a system fail to do that?
- The one thing you still don't fully understand about RAG.
After filling in UNDERSTANDING.md, use the AI mentor pattern in docs/meta/learning-flow.md to get feedback. Record the feedback in UNDERSTANDING_FEEDBACK.md.
6. Project Assignment
See PROJECT.md and source/project.md for the full specification including file structure and extended requirements.
Core Requirement
Build a PDF research assistant in code/ that answers questions about a document with grounded, cited answers and a faithfulness check:
chunker.py— split document text into overlapping, id-tagged chunks. Learner core.retriever.py— build a Chroma index of the chunks and retrieve top-k. (Provided — your Project 03 retriever, reused.)generator.py— construct the grounded prompt (context + ids + refusal instruction) and produce an answer with citations. Learner core.faithfulness.py— decompose the answer into claims, verify each against the context, computeF = |V|/|S|. Learner core.rag.py— the end-to-end pipeline wiring it together. (Provided orchestrator.)
Extended: char/page citations; compare two chunking strategies; add context-relevance scoring.
Documents are split into overlapping, id-tagged chunks; the generator answers only from context, cites the chunk ids, and refuses when the answer is absent; faithfulness.py reports F = |V|/|S| as a number; the guiding tests pass; UNDERSTANDING.md done before any code; FAILURE_ANALYSIS.md has ≥3 experiments; EVALUATION.md is quantitative; STARCALLOS_REFLECTION.md names ≥1 concrete pattern.
Open code/README.md for setup, the milestone build order, and the file roles (which files are provided vs. learner-owned). Run python -m pytest to see the failing guiding tests, then implement the learner-owned functions in milestone order until they pass.
7. Project Milestones
Work through these in order. Each milestone should produce runnable code before moving on.
M1 — Chunk
chunker.py splits text into overlapping, id-tagged chunks. Validation: adjacent chunks overlap by overlap; every chunk has a stable id + source.
M2 — Index & Retrieve
Build a Chroma index of chunks; retrieve top-k for a query (Project 03 reuse). Validation: a question retrieves the chunk that actually contains the answer.
M3 — Grounded answer
generator.build_prompt stuffs context + ids and instructs answer-only-from-context. Validation: the prompt contains the chunks, their ids, and an explicit refusal instruction.
M4 — Citations
answer() attaches the supporting chunk id(s) to the answer. Validation: each claim points to the chunk it came from.
M5 — Faithfulness
faithfulness.py decomposes the answer into claims and scores F = |V|/|S|. Validation: a grounded answer scores ~1.0; an injected false claim drops the score.
M6 — Break / Evaluate
Ask an out-of-document question; force a hallucination; measure the drop. Validation: off-doc question → refusal; faithfulness number recorded in FAILURE_ANALYSIS.md.
8. Self-Evaluation
After building, honestly evaluate your implementation against these criteria. Record your answers in EVALUATION.md.
| Criterion | Does your implementation... | Pass? |
|---|---|---|
| Real chunking | split documents into overlapping, id-tagged chunks (not one vector per doc)? | ☐ |
| Retrieval reuse | build a Chroma index and retrieve top-k by meaning? | ☐ |
| Actually grounded | instruct the model to answer only from context, and pass the chunks in the prompt? | ☐ |
| Refuses | answer "I don't know" when the document doesn't contain the answer? | ☐ |
| Cites | trace each claim back to the supporting chunk id/source? | ☐ |
| Measures faithfulness | compute F = |V|/|S| over the answer's claims, as a number? | ☐ |
Your implementation may have problems if:
- The model answers off-document questions confidently instead of refusing.
- You never pass the retrieved chunks into the prompt (the model is answering from memory).
- Answers have no citations, so a human can't verify them.
- You report "it works" with no faithfulness number and no broken-case test.
- Chunks have no stable id/source, so you can't cite or verify against them.
9. Common Mistakes
| Mistake | Why It Happens | Consequence | Fix |
|---|---|---|---|
| No grounding instruction (or a weak one) | Assuming "context in prompt" is enough | Model answers from its own memory; fluent but ungrounded | System message: "answer ONLY from the context; if absent, say you don't know" |
| Chunks too large | "More context is better" | Diffuse embeddings; retrieval can't find precise matches; answer buried in noise | Smaller fixed-size chunks; iterate |
| Chunks too small / no overlap | Maximizing precision | A fact spanning a boundary is unfindable | Add ~10–20% overlap so boundary facts survive |
| Never refusing | No "I don't know" path in the prompt | Confident hallucination on off-document questions | Explicit refusal instruction + test it on an off-doc question |
| No citations | Treating the answer as the deliverable | Unverifiable answers — useless in legal/medical/finance | Tag chunks with ids; have the model cite the id per claim |
| "It works" with no faithfulness check | Eyeballing a few good answers | Hallucinations ship silently | Score F = |V|/|S|; verify each claim against context |
| Mixing embedding models | Index and query embedded differently | Retrieval returns garbage; whole pipeline fails silently | Pin one embedding model for chunks and query |
10. Connections
Builds On
Project 03 built retrieval — index chunks, query top-k. Project 04 is that retriever feeding an LLM (Project 01's chat completion), with chunking in front and grounding/citation/faithfulness around it. The "same embedding model both sides" rule and the persistent Chroma index come straight from Project 03.
Enables
Project 05 (memory) is RAG over your own past instead of a PDF — chunk and index conversations/notes, retrieve relevant memories, ground responses in them. Project 07 (evaluation framework) generalizes the faithfulness pass here into a full LLM-as-judge eval suite.
Production Patterns
The deployed pattern is this pipeline plus guardrails: chunk (often recursive/semantic), retrieve top-k (often + re-rank from Project 03), ground with a strict system prompt, cite with verifiable pointers, and monitor faithfulness offline and online. Real systems prompt-cache the document context to cut cost.
StarcallOS Relevance
Any StarcallOS feature that answers from your documents, notes, or history — "what did I decide about X," "what does this contract say about termination" — is RAG. The disciplines built here decide whether StarcallOS gives a grounded, cited answer you can trust or a confident hallucination: chunk well, ground hard, cite the source, and refuse when the answer isn't there.
Sources
See source/resources.md for the complete annotated source list.
Tier 1 — Official Documentation
sources/official-docs/anthropic-citations.md— claim → source location (char_location/page_location),cited_text, why API citations beat prompt-asked quotessources/official-docs/chromadb.md— the retrieval index (carried from Project 03)
Tier 2 — Foundational Papers
sources/papers/rag-paper.md— Lewis et al. 2020: RAG = parametric + non-parametric memory, provenance, updatable knowledgesources/papers/ragas.md— Es et al. 2023: faithfulness (the hallucination metric), answer/context relevancesources/papers/sentence-bert.md— bi-encoder embeddings behind retrieval (carried)
Tier 3 — Engineering Guides
sources/articles/chunking-strategies.md— Pinecone: chunk-size tradeoff, overlap, fixed vs recursive vs semanticsources/articles/sbert-retrieve-rerank.md— retrieve-then-rerank when first-stage retrieval isn't precise enough
Tier 4 — Educational Sources
- None specific to this lesson — the Project 02–03 embedding/retrieval sources are the prerequisites.