# Project 02: Token & Embedding Explorer

# source/project.md — Detailed Project Specification

> Generated by: skills/lesson-generator
> For the high-level overview see the root PROJECT.md and the teaching content in
> `source/lesson.agent.md`.
> This file contains implementation contracts, not teaching content.

---

## Definition of Done

The project is complete when all of the following are true:

- [ ] Core implementation runs without errors (all four modules)
- [ ] All project milestones M1–M6 demonstrated (see `lesson.agent.md` §7)
- [ ] `cosine_similarity` is implemented **by hand** (no library helper for the core formula)
- [ ] UNDERSTANDING.md completed before first line of code
- [ ] FAILURE_ANALYSIS.md contains at least 3 intentional experiments (one must be an analogy that fails)
- [ ] EVALUATION.md contains quantitative results — real cosine numbers and token counts, not impressions
- [ ] STARCALLOS_REFLECTION.md identifies at least one concrete applicable pattern
- [ ] Guiding tests in `code/tests/` pass (`python -m pytest`)

---

## File Specification

Four modules in `code/`. Each names a single core function the learner owns; the runnable
`main()` harness is provided so the only friction is the concept, not the plumbing.

### `tokenizer_explorer.py` — `partial`

**Purpose:** Make tokenization visible — turn text into `tiktoken` IDs, show the text chunk each
ID maps to, count tokens, and prove the round-trip is lossless. (Milestone M1.)

**Key functions (learner-owned core):**

```python
def encode(text: str) -> list[int]:
    """Return the list of integer token IDs for `text` under ENCODING_NAME."""

def decode(ids: list[int]) -> str:
    """Inverse of encode: token IDs back to the exact original text."""

def token_pieces(ids: list[int]) -> list[str]:
    """The decoded text chunk for each individual ID (so boundaries are visible)."""

def count(text: str) -> int:
    """Token count — equal to len(encode(text)); this is what you are billed on."""
```

**Provided:** `ENCODING_NAME = "o200k_base"`, `get_encoder()`, and `main()` (formats the
inspection table and prints the round-trip check).

**Dependencies:** `tiktoken`, `config.py`.

**Does not:** embed text, compute similarity, or call any network API. Tokenization is a local,
deterministic codec.

### `embedding_explorer.py` — `partial`

**Purpose:** Turn text into a meaning-bearing vector via `litellm`, and read the vector's
dimensionality **from the vector itself** (never hardcoded). (Milestone M3.)

**Key functions (learner-owned core):**

```python
def embed(text: str) -> np.ndarray:
    """Embed `text` with default_embedding_model() and return the vector as a numpy array."""
```

**Provided:** `main()` (embeds a sample, prints `len(vector)` and the first few components).

**Dependencies:** `litellm`, `numpy`, `config.py` (`default_embedding_model`).

**Does not:** choose the provider (that is `config.py`'s job) or hardcode a dimension.

### `similarity_calculator.py` — `learner`

**Purpose:** The conceptual core of the project — implement cosine similarity from scratch and use
it to score text pairs. (Milestone M4.)

**Key functions (learner-owned core):**

```python
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    """cos(a,b) = (a·b) / (‖a‖·‖b‖), computed by hand. Range [-1, 1]. No library shortcut."""
```

**Provided:** `main()` (embeds a paraphrase pair and an unrelated pair, prints both scores).

**Dependencies:** `numpy`, `embedding_explorer.embed`.

**Does not:** call `sklearn.metrics.pairwise.cosine_similarity` or any prebuilt cosine helper —
the formula is the learning target.

### `corpus_search.py` — `learner`

**Purpose:** Embed a small corpus once, then rank it against a query by cosine — a minimal semantic
search, the engine of Projects 03–04. (Milestone M5.)

**Key functions (learner-owned core):**

```python
def search(query: str, corpus: list[str], k: int = 3) -> list[tuple[str, float]]:
    """Embed query + each corpus item, score with cosine_similarity, return the top-k
    (text, score) pairs sorted by score descending."""
```

**Provided:** a sample `CORPUS`, and `main()` (runs a sample query and prints ranked results).

**Dependencies:** `embedding_explorer.embed`, `similarity_calculator.cosine_similarity`.

**Does not:** persist embeddings, use a vector database, or re-embed the corpus per query inside a
loop when it can embed once (that optimization is the bi-encoder lesson — note it, then keep it
simple here).

---

## Input / Output Contracts

> The observable behavior the implementation must satisfy.

| Function | Input | Expected Output | Error Behavior |
|----------|-------|-----------------|----------------|
| `encode(text)` | `str` | `list[int]` of token IDs | empty string → `[]` |
| `decode(ids)` | `list[int]` | `str`; `decode(encode(s)) == s` | invalid ID → underlying `tiktoken` error |
| `count(text)` | `str` | `int == len(encode(text))` | — |
| `embed(text)` | `str` | `np.ndarray`, shape `(d,)`, `d` model-specific (768 for nomic) | provider/network error surfaces (do not swallow) |
| `cosine_similarity(a, b)` | two `np.ndarray` same length | `float` in `[-1, 1]`; `cos(v,v)=1`, `cos(v,-v)=-1` | mismatched lengths → raise (do not pad) |
| `search(query, corpus, k)` | `str`, `list[str]`, `int` | `list[(text, score)]` length `min(k, len(corpus))`, sorted desc | empty corpus → `[]` |

---

## Extended Requirements

Beyond the core implementation — complete these after the core is working.

- [ ] Reproduce `king − man + woman ≈ queen` in `corpus_search` or an experiment script, and find
      **one analogy triple that fails** (record it in FAILURE_ANALYSIS.md — this is required, not a bug).
- [ ] Compare token counts of the *same* text under two encodings (e.g. `o200k_base` vs `cl100k_base`)
      and explain why they differ.
- [ ] (Optional) Swap the embedding model (local ↔ OpenAI via `.env`) and observe the dimensionality
      change; confirm you **cannot** meaningfully compare vectors across models.

---

## Known Difficulty Spikes

Areas where learners typically get stuck. Listed so the learner expects them, not so they can avoid them.

1. **Forgetting to divide by the norms** in cosine → values outside `[-1, 1]`. The guiding tests
   (`cos(v,v)=1`, `cos(v,-v)=-1`, bounds) catch this immediately.
2. **Hardcoding the embedding dimension** (copying `1536` from a tutorial) → breaks the moment the
   default provider is `nomic-embed-text` (768). Read `len(vector)`.
3. **Re-embedding the query inside the ranking loop** in `corpus_search` → correct but wasteful;
   embed the query once, then each corpus item once.
4. **First `tiktoken` run needs to fetch the encoding file** (one-time network/cache). After that it
   is offline. Tokenizer tests may fail on a cold machine with no network until the encoding caches.

---

## Debugging Approach

When things break, check in this order:

1. Environment — is `.env` loaded? For embeddings, is a provider available (`USE_OLLAMA=1` with Ollama
   running, or a cloud key)?
2. Inputs — print the raw text before encoding/embedding.
3. Outputs — print the raw return: `enc.encode(text)` and `r["data"][0]["embedding"]` before parsing.
4. Isolation — does `cosine_similarity` pass its offline tests? If yes, the bug is in embedding/ranking,
   not the math.
5. Source — re-read the relevant section of `source/lesson.agent.md` (BPE mechanics §3, cosine math §3).

---

## Integration Notes

> How this project connects to others in the curriculum.

**Depends on:** Project 01 (provider abstraction via `config.py`/LiteLLM; the per-token cost model that
this project's tokenizer makes concrete).

**Depended on by:** Project 03 (semantic search — `search()` here is the seed of it), Project 04
(RAG/PDF assistant — embed + cosine retrieval), Project 05 (memory — recall by embedding similarity).
`cosine_similarity` and the "embed once, compare many" pattern carry forward unchanged.