# Hugging Face — Summary of the Tokenizers (BPE, WordPiece, Unigram)

**Type:** official-doc
**Tier:** 1 (Official Doc)
**Author(s):** Hugging Face (Transformers documentation)
**Date:** Accessed 2026-06-01
**URL:** https://huggingface.co/docs/transformers/en/tokenizer_summary
**Accessed:** 2026-06-01

## Why This Source Matters

This is the authoritative, worked-example explanation of *how subword tokenization algorithms actually build their vocabularies*. Where `tiktoken-bpe.md` gives the API, this source gives the **mechanism**: it walks BPE merge-by-merge on a tiny corpus, so the learner can reproduce the algorithm by hand. It also establishes the word-vs-character-vs-subword tradeoff that motivates tokenization in the first place — the conceptual spine of Project 02's tokenization half.

## Key Claims

### The motivating tradeoff (why subword)
- **Word-level**: vocabulary becomes enormous (every inflection — `love`, `loving`, `loved`, `lovingly` — is its own token); huge embedding matrix; new/unseen words map to `<unk>` and are unrepresentable.
- **Character-level**: tiny vocabulary, every word representable, no `<unk>` — but **sequences get very long** and a single char (`"l"`) carries far less meaning than a word (`"love"`), hurting performance.
- **Subword** (the compromise): split between words and characters. Keep common words whole; decompose rare words into known subwords. Compact vocabulary *and* the ability to represent unseen words. Example: `annoyingly` → `["annoying", "ly"]` or `["annoy", "ing", "ly"]`.

### Byte Pair Encoding (BPE), step by step
1. A **pre-tokenizer** splits text into words + frequencies, e.g. `("hug",10), ("pug",5), ("pun",12), ("bun",4), ("hugs",5)`.
2. Build a **base vocabulary** from all characters: `["b","g","h","n","p","s","u"]`.
3. **Iteratively merge the most frequent adjacent pair.** `"u"+"g"` is most frequent → merge into `"ug"`, add to vocab.
4. Next most frequent pair `"u"+"n"` → merge into `"un"`.
5. Continue until the **target vocabulary size** is reached. **Final vocab size = base size + number of merges.** Example: original GPT uses BPE with vocab 40,478 = 478 base + 40,000 merges.
- BPE is **deterministic**: a fixed list of merge rules applied in order.

### Byte-level BPE (what GPT-2 / tiktoken use)
- Using all Unicode chars as the base would be huge. **Byte-level BPE uses the 256 byte values as the base vocabulary**, so *every* string is tokenizable with **no `<unk>` token ever**.
- GPT-2 vocab = 50,257 = 256 byte tokens + 50,000 merges + 1 end-of-text token.

### Contrast algorithms (for accurate mental model)
- **WordPiece** (BERT family): merges the pair that most increases training-data likelihood — `score(u,g) = freq(ug) / (freq(u)·freq(g))` — i.e. how *informative* the merge is, not merely how frequent. BPE merges the most frequent pair; WordPiece merges the most surprising-together pair.
- **Unigram** (T5, Pegasus): starts with a *large* candidate vocab and **removes** the lowest-loss tokens until target size; probabilistic, can produce multiple tokenizations and pick the highest-probability one.
- **SentencePiece**: applies BPE/Unigram to **raw bytes including the space char** (`▁`), so it works for languages without spaces (Chinese, Japanese).

## Relevant To

- concepts: [tokenization, byte-pair-encoding, subword-tokenization, vocabulary, wordpiece, unigram]
- projects: [02-token-embedding-explorer]

## Notes

- The tiny `hug/pug/pun/bun/hugs` corpus is ideal for a learner exercise: have them perform 2–3 BPE merges by hand, then confirm against `tiktoken` behavior on real text.
- Key correction to a common misconception: BPE merges are driven by **frequency in a training corpus**, not by linguistic morphology. Subword boundaries often *look* morphological but are a statistical artifact.