Token & Embedding Explorer

Learning Objectives

By the end of this project, you will be able to:

• Explain what a token is and why LLMs operate on tokens instead of words or characters.
• Run a real BPE tokenizer (tiktoken), inspect token boundaries and integer IDs, and predict roughly how many tokens text will cost before calling an API.
• Reproduce two or three BPE merge steps by hand and explain why merges are driven by corpus frequency, not grammar.
• Explain what an embedding vector is — what the numbers represent and why "meaning becomes geometry."
• Compute cosine similarity by hand and explain why it measures angle, not distance or magnitude.
• Explain why king − man + woman ≈ queen works in embedding space — and demonstrate a case where analogy arithmetic fails.
• Distinguish a tokenizer from an embedding model, and a word embedding from a sentence embedding (bi-encoder).
• Choose an appropriate embedding model and reason about why its dimensionality (e.g. 768 vs 1536) is model-specific.

1. Motivation

Why This Exists

Computers process numbers, not text. Two distinct numeric conversions stand between raw text and a working AI system: tokenization (text → integer IDs the model can read) and embedding (text → a vector that encodes meaning). Both predate modern LLMs — BPE began as a compression technique, and dense word vectors were popularized by word2vec in 2013 — but they are the unglamorous machinery every AI engineer must understand to debug real systems.

What Breaks Without It

Mismatched embedding models, wrong chunk sizes, failed similarity searches, surprise token bills. If you embed a query with one model and your documents with another, similarity scores become meaningless and retrieval silently returns garbage — with no error raised.

Real-World Stakes

Every semantic search, RAG pipeline, recommendation engine, and deduplication system rests on these two conversions. Misjudge tokenization and you blow the context window or the budget; mismatch embedding models and retrieval quietly fails. This connects directly to Project 01's token-cost work and forward to Projects 03–04.

2. Mental Model

Explain Like I'm 12

Imagine you have to mail a long book, but the post office only accepts numbered LEGO bricks. First you chop the book into common little pieces ("ing", "the", "pre") and give each piece a brick number — that's tokenization. Now imagine a magic map of a city where every word lives at an address, and words that mean similar things live on the same street. "King" and "queen" are neighbors; "banana" is across town. The address of a word is its embedding. To ask "are these two words similar?" you don't measure how far apart the houses are — you stand at the city center and check whether they're in the same direction. That "same direction?" check is cosine similarity.

Explain Like I'm a Software Engineer

• Tokenization is a lossless, reversible str ⇄ list[int] codec with a learned dictionary. BPE builds that dictionary by starting from bytes and greedily merging the most frequent adjacent pair until it hits a target vocabulary size. len(enc.encode(text)) is your token count; on average one token ≈ 4 bytes of text.
• Embedding is a function str → R^d (d = 768, 1536, …) learned so that semantically related inputs map to vectors pointing in similar directions. word2vec established this for words; SBERT extended it to whole sentences with a fixed-size output vector.
• Cosine similarity is the L2-normalized dot product, x·y / (‖x‖‖y‖) — the cosine of the angle between two vectors, ranging −1…1.

Real-World Analogy

How It Works (Diagram)

  "unbelievable"
        │ tokenizer (BPE)            │ embedding model
        ▼                            ▼
  [un][bel][iev][able]        [ 0.02, 0.41, -0.13, ... , 0.07 ]   (d numbers)
  4 integer IDs                       a point/direction in R^d

  cosine( vec("unbelievable"), vec("incredible") )  →  ~0.7   (close direction = similar)
  cosine( vec("unbelievable"), vec("granite")     )  →  ~0.1   (near-orthogonal = unrelated)

3. Technical Explanation

Formal Definition

BPE tokenization: given a target vocab size V, learn an ordered list of merge rules over a base alphabet; at encode time, apply merges greedily to map a string to a sequence of token IDs. Final vocab size = base size + number of merges.

Embedding: a learned map E: text → R^d. Cosine similarity cos(x,y) = (x·y)/(‖x‖‖y‖) measures the angle between two embeddings, in [−1, 1].

How It Works Step by Step (BPE by hand)

Start from a corpus of word→frequency, each word split into characters:

("h" "u" "g", 10) ("p" "u" "g", 5) ("p" "u" "n", 12) ("b" "u" "n", 4) ("h" "u" "g" "s", 5)

1. The most frequent adjacent pair is u·g (in hug, pug, hugs) → merge to ug.
2. Next is u·n (in pun, bun) → merge to un.
3. The vocabulary grows [b,g,h,n,p,s,u] → […, ug, un]. Continue until the target size.

The merges look morphological but are pure frequency statistics. Byte-level BPE (what GPT-2 / tiktoken use) starts from the 256 byte values, so every string is tokenizable and there is no <unk> token, ever.

Key Concepts

4. Guided Examples

The lab stack: tiktoken (tokenization), litellm (embeddings — provider chosen by config.py, default local ollama/nomic-embed-text at 768-dim), and numpy (cosine).

Example 1: Simplest Case — see the tokens

import tiktoken

enc = tiktoken.get_encoding("o200k_base")          # GPT-4o-family encoding
ids = enc.encode("Tokenization is not magic.")
print(ids)                                          # e.g. [2350, 2860, ...] integer IDs
print([enc.decode([i]) for i in ids])               # the text chunk each ID maps to
print("token count:", len(ids))                     # what you'd be billed on
print("round-trip:", enc.decode(ids))               # lossless: exact original text

Example 2: Real-World Case — meaning by cosine

import numpy as np, litellm
from config import default_embedding_model

def embed(text: str) -> np.ndarray:
    r = litellm.embedding(model=default_embedding_model(), input=[text])
    return np.array(r["data"][0]["embedding"])

def cosine(a, b) -> float:
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

a = embed("the cat sat on the mat")
b = embed("a feline rested on the rug")   # paraphrase
c = embed("quarterly tax filing")          # unrelated
print(round(cosine(a, b), 3))   # high: same meaning, different words
print(round(cosine(a, c), 3))   # low: unrelated topics
print("dims:", a.shape[0])      # 768 for nomic-embed-text — model-specific

Example 3: When It Fails — analogy arithmetic

king, man, woman = embed("king"), embed("man"), embed("woman")
queen, banana = embed("queen"), embed("banana")
analogy = king - man + woman
print("→ queen :", round(cosine(analogy, queen), 3))    # expected: relatively high
print("→ banana:", round(cosine(analogy, banana), 3))   # expected: low
# Now try a pair the model has weak signal for and watch the analogy degrade.

5. Reflection Before Building

After filling in UNDERSTANDING.md, use the AI mentor pattern in docs/meta/learning-flow.md to get feedback. Record the feedback in UNDERSTANDING_FEEDBACK.md.

6. Project Assignment

See PROJECT.md for the full project specification including file structure and extended requirements.

Core Requirement

Build the explorer in code/ — four modules:

• tokenizer_explorer.py — given text, show each token's boundary, its integer ID, and the total count; demonstrate the lossless round-trip.
• embedding_explorer.py — embed text via litellm and report the vector's dimensionality and a few components.
• similarity_calculator.py — implement cosine similarity by hand with numpy (no library shortcut for the core function) and score pairs of texts.
• corpus_search.py — embed a small corpus, then return the nearest neighbors of a query by cosine.

Extended: reproduce king − man + woman and find one analogy that fails; compare token counts of the same text under two encodings; (optional) swap the embedding model and confirm you cannot compare vectors across models.

7. Project Milestones

Work through these in order. Each milestone should produce runnable code before moving on.

8. Self-Evaluation

After building, honestly evaluate your implementation against these criteria. Record your answers in EVALUATION.md.

9. Common Mistakes

10. Connections

Builds On

Project 01 billed you per token — this lesson opens that black box (tiktoken) so token count and cost stop being mysterious. The provider-abstraction layer (LiteLLM / config.py) you met in Project 01 is reused here for embeddings.

Enables

Embeddings + cosine are the literal engine of Project 03 (semantic search) and Project 04 (RAG / PDF assistant) — both retrieve by embedding a query and ranking documents by cosine. SBERT's "encode once, compare many" property is what makes that scalable. Project 05 (memory) stores and recalls by embedding similarity.

Production Patterns

Real systems separate a bi-encoder (embed each item independently → fast retrieval) from a cross-encoder (re-encode a pair → highest accuracy, too slow at scale), and combine them: retrieve with the bi-encoder, re-rank the top-k with a cross-encoder. Vectors are stored in a vector database (Project 03).

StarcallOS Relevance

Sources

See source/resources.md for the complete annotated source list.