Key Points
- 1.A two-hour first-principles build of the GPT-style byte-pair-encoding (BPE) tokenizer in Python.
- 2.Karpathy is direct: tokenization is his least favorite part of LLMs, and most 'model bugs' you encounter are actually tokenizer bugs.
- 3.Implements the BPE training algorithm, vocabulary building, and the encode/decode round-trip - then compares to Tiktoken (GPT) and SentencePiece (Llama).
- 4.Walks through real-world failure cases - SolidGoldMagikarp, arithmetic, code, non-English text - and explains exactly which tokenizer choice causes each.
Summary
Why tokenization is the source of weird LLM behavior
Many failures people blame on 'the model' are actually tokenizer artifacts. Non-Latin scripts get split into many more tokens than English (so context costs more and quality drops), numbers tokenize inconsistently which is why arithmetic is shaky, trailing whitespace silently changes everything, and certain rare token IDs (the famous SolidGoldMagikarp) trigger broken behavior because they appeared in the tokenizer's training corpus but not in the model's training corpus.
Building BPE end to end
Starts from raw UTF-8 bytes, builds the merge algorithm by hand, walks through vocabulary construction iteration by iteration, and shows the exact data structure choices. The encode/decode round-trip is implemented in plain Python so you can step through it. Karpathy emphasizes that the merge order is the entire 'learned' state - there is no neural network in the tokenizer.
GPT-style vs Llama-style tokenizers
Side-by-side comparison with Tiktoken (used by GPT-2/3/4) and SentencePiece (used by Llama). The differences come down to how each handles pre-tokenization splits, byte fallback for unknown characters, and special tokens. Karpathy shows the byte-for-byte equivalence and where each toolkit makes a different trade-off.
Special tokens and their pitfalls
Adding special tokens after training (chat markers, FIM tokens, system prompts) requires surgery on the tokenizer state. Doing it wrong silently breaks the model; doing it right is straightforward but undocumented. He shows the exact code path.
Practical takeaways for builders
Always inspect what the tokenizer is doing on your real data before debugging the model. For multilingual or code-heavy use cases, pick the tokenizer first and the model second. For new languages or domains, BPE you train yourself can outperform a frontier model's tokenizer on token efficiency by 2-3x.
Why this matters for prompt engineering
Karpathy connects tokenization to prompt design - why 'count the letter r in strawberry' fails, why JSON output is more reliable when keys are short, and why instruction-following is sensitive to leading whitespace. The recurring theme: the model only ever sees tokens, never characters.
Worth watching for
Engineers and researchers who want to deeply understand LLM internals - and anyone who has ever been confused by why their prompt 'almost worked'.
- llms
- tokenization
- fundamentals
- from-scratch