# Project 03: Semantic Search

# source/project.md — Detailed Project Specification

> Generated by: skills/lesson-generator
> For the high-level overview see the root PROJECT.md and the teaching content in
> `source/lesson.agent.md`.
> This file contains implementation contracts, not teaching content.

---

## Definition of Done

The project is complete when all of the following are true:

- [ ] Core implementation runs without errors (indexer, semantic_search, baseline, evaluate)
- [ ] All project milestones M1–M6 demonstrated (see `lesson.agent.md` §7)
- [ ] The index is a **persistent Chroma collection** (not an in-memory Python list)
- [ ] Distances are correctly converted to similarities (no ranking inversion)
- [ ] `evaluate.py` reports **recall@k** of the ANN index vs. the exact brute-force baseline (a number)
- [ ] UNDERSTANDING.md completed before first line of code
- [ ] FAILURE_ANALYSIS.md contains ≥3 intentional experiments (e.g. metric swap, `ef_search` sweep, forced inversion)
- [ ] EVALUATION.md contains quantitative results (recall@k, real distances/similarities), not impressions
- [ ] STARCALLOS_REFLECTION.md identifies at least one concrete applicable pattern
- [ ] Guiding tests in `code/tests/` pass (`python -m pytest`)

---

## File Specification

Five modules in `code/`. Each names the core the learner owns; the runnable harness and the embedding
plumbing are provided so the friction is the retrieval concept, not setup.

### `embedding_helpers.py` — `provided`

**Purpose:** Wrap `litellm` so the rest of the project gets embeddings as plain Python lists/arrays
without repeating the provider call. Provided complete so the learner focuses on retrieval, not API glue.

**Key functions:** `embed_many(texts: list[str]) -> list[list[float]]`, `embed_one(text) -> list[float]`.

**Does not:** choose the provider (that is `config.py`) or compute similarity.

### `indexer.py` — `partial`

**Purpose:** Build a **persistent** Chroma collection from the corpus. (Milestone M1.)

**Key function (learner-owned core):**

```python
def build_index(docs: list[str], ids: list[str], metadatas: list[dict] | None = None,
                collection_name: str = "corpus") -> "chromadb.Collection":
    """Create-or-get a PersistentClient collection with space='cosine', embed docs via
    embedding_helpers.embed_many, and add them. Idempotent: re-running must NOT duplicate
    records (upsert, or skip ids already present)."""
```

**Provided:** `PERSIST_DIR` constant, `get_client()` (returns a `PersistentClient`), and `main()` that
builds the index from the sample corpus and prints `collection.count()`.

**Does not:** answer queries or compute recall.

**Example I/O:**
```text
build_index(["the cat sat", "tax policy"], ["d0","d1"])  →  collection with count() == 2
# run build_index again with the same ids → count() still 2 (no duplicates)
```

### `semantic_search.py` — `learner`

**Purpose:** The core — query the collection and return ranked results with **distance converted to
similarity**. (Milestones M2, M3.)

**Key functions (learner-owned core):**

```python
def to_similarity(distance: float) -> float:
    """Chroma space='cosine' returns cosine DISTANCE in [0,2]; return 1 - distance."""

def search(collection, query: str, k: int = 5) -> list[tuple[str, float, dict]]:
    """Embed `query`, call collection.query(query_embeddings=[qv], n_results=k), and return
    [(document, similarity, metadata)] sorted by similarity DESCENDING (== distance ascending)."""
```

**Provided:** `main()` that builds/loads the index and prints ranked results for a sample query.

**Does not:** re-embed the whole corpus per query (the index already holds the vectors).

**Example I/O:**
```text
search(col, "monetary policy and the economy", k=2)
→ [("The central bank raised interest rates this quarter.", 0.71, {...}),
   ("Inflation eroded household savings.",                  0.66, {...})]
# similarities DESCENDING; the finance doc outranks lexically-similar-but-unrelated text
```

### `baseline.py` — `partial`

**Purpose:** Exact brute-force cosine ranking (reuse Project 02's cosine) to validate the ANN index, plus
a recall helper. (Milestones M4, M5.)

**Key functions (learner-owned core):**

```python
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    """Cosine from scratch (carry over from Project 02). No library shortcut."""

def exact_rank(query_vec, doc_vecs: list[np.ndarray], k: int) -> list[int]:
    """Return the indices of the top-k docs by exact cosine, similarity descending."""

def recall_at_k(approx_ids: list[int], exact_ids: list[int], k: int) -> float:
    """|approx top-k ∩ exact top-k| / k, in [0, 1]."""
```

**Provided:** `main()` demoing the baseline on the sample corpus.

**Example I/O:**
```text
recall_at_k([0, 2, 5], [0, 2, 9], k=3)  →  0.6667   (two of three overlap)
exact_rank(qv, doc_vecs, k=2)           →  [3, 7]    (indices, best first)
```

### `evaluate.py` — `learner`

**Purpose:** Tie it together — run several queries through both the Chroma ANN path and the exact
baseline, and report recall@k. (Milestone M5.)

**Key function (learner-owned core):**

```python
def evaluate(collection, queries: list[str], k: int = 5) -> dict:
    """For each query: get ANN top-k ids from `collection`, get exact top-k ids from baseline over the
    same vectors, compute recall_at_k. Return {'per_query': {...}, 'mean_recall': float}."""
```

**Provided:** sample `QUERIES`, and `main()` printing a small recall table.

**Example I/O:**
```text
evaluate(col, ["interest rates", "magic school"], k=3)
→ {"per_query": {"interest rates": 1.0, "magic school": 1.0}, "mean_recall": 1.0}
# tiny corpus → ANN matches exact perfectly; the method is what matters
```

### `rerank.py` — `learner` (Extended / optional)

**Purpose:** Two-stage retrieve-then-rerank. Retrieve top-k from Chroma, re-score with a cross-encoder,
return the reordered list. (Milestone M6, extended.)

```python
def rerank(query: str, candidates: list[str]) -> list[tuple[str, float]]:
    """Score each (query, candidate) pair with a cross-encoder and return candidates sorted by
    score descending. Only ever called on the small candidate set from search(), never the full corpus."""
```

---

## Input / Output Contracts

| Function | Input | Expected Output | Error Behavior |
|----------|-------|-----------------|----------------|
| `build_index(docs, ids)` | `list[str]`, `list[str]` | Chroma collection, `count() == len(set(ids))` | re-run with same ids → no duplicates |
| `to_similarity(d)` | cosine distance `float` in `[0,2]` | similarity `1 - d` in `[-1,1]` | — |
| `search(col, q, k)` | collection, `str`, `int` | `list[(doc, similarity, meta)]`, len ≤ k, similarity descending | empty collection → `[]` |
| `cosine_similarity(a,b)` | two `np.ndarray` | `float` in `[-1,1]`; `cos(v,v)=1` | mismatched lengths → raise |
| `recall_at_k(approx, exact, k)` | two `list[int]`, `int` | `float` in `[0,1]` | k=0 → define as 1.0 or raise (document choice) |
| `evaluate(col, queries, k)` | collection, `list[str]`, `int` | `{'per_query':{...}, 'mean_recall':float}` | empty queries → mean_recall over 0 → 1.0 or raise |

---

## Extended Requirements

- [ ] `rerank.py`: cross-encoder re-ranking of the top-k; report how the top result changed.
- [ ] Compare `space="cosine"` vs `space="l2"` on the same corpus; explain the ranking differences.
- [ ] Sweep `ef_search` (e.g. 10, 50, 100) and record recall@k vs. qualitative latency.

---

## Known Difficulty Spikes

1. **Distance vs. similarity inversion** — Chroma returns distances (lower = closer); Project 02 returned
   similarities (higher = closer). Sorting the wrong direction returns the *worst* matches with no error.
2. **Forgetting `space="cosine"`** — the default is `l2`; for text embeddings rankings degrade subtly.
3. **Idempotent indexing** — naive `add` on re-run duplicates records; use `upsert` or skip existing ids.
4. **Mapping Chroma ids back to baseline indices** for recall@k — keep a stable id↔index scheme.
5. **First Chroma run / default embedding model download** — pin the lab's `litellm` embeddings via
   `embedding_helpers` and pass `query_embeddings` so Chroma does not pull its own default model.

---

## Debugging Approach

1. Environment — is `.env` loaded? Is an embedding provider available (`USE_OLLAMA=1` or a key)?
2. Inputs — print the query and the first doc before embedding.
3. Outputs — print the raw `collection.query(...)` dict: look at `ids` and `distances` directly.
4. Direction — sanity-check: the best result must have the **smallest** distance / **largest** similarity.
5. Isolation — does `baseline.cosine_similarity` pass its offline tests? If yes, a ranking bug is in the
   Chroma path or the distance→similarity conversion, not the math.
6. Source — re-read `source/lesson.agent.md` §3 (distance↔similarity) and `sources/official-docs/chromadb.md`.

---

## Integration Notes

**Depends on:** Project 02 (embeddings; `cosine_similarity` reused as the exact baseline; the
same-model-both-sides rule).

**Depended on by:** Project 04 (RAG/PDF assistant — this search step feeds retrieved chunks to the LLM),
Project 05 (memory — recall by vector-DB lookup). The retrieve-then-rerank pattern recurs throughout.