What You Will Learn
BEGINNER
- What tokens actually are (and why "words" is the wrong mental model)
- BPE: the algorithm GPT uses to split words into chunks
- Why tokenization directly affects your cost, latency, and code quality
The Big Misconception: LLMs Don't Read Words
Here's something that surprises every newcomer: LLMs don't process words. They don't process characters either. They process tokens — small chunks of text that might be a whole word, part of a word, a single character, or even punctuation. The model has a fixed vocabulary of tokens (GPT-4's tokenizer knows about 100,000 different tokens), and every input you send is first split into these tokens.
Think of tokens as the "letters" of the LLM's alphabet, except this alphabet has 100,000 symbols instead of 26. The word unbelievable might be split into un + believe + able. The word hamburger might be ham + burger. Or it might be one token. It depends on how often the model saw that exact word during training.
Why Not Just Words?
You might wonder: why not just split on spaces and call each word a token? Three reasons:
- Vocabulary size: English alone has ~1 million words; add names, technical terms, misspellings, and you'd need a vocab of millions. With 100K tokens, the model is manageable.
- Sub-word understanding: By splitting
unbelievableintoun+believe+able, the model learns thatun-means "not" and-ablemeans "capable of". This generalizes to words it has never seen (unprogrammable→ it can figure out). - Multi-language support: Tokenizers can mix English, Chinese, code, emoji — all in the same vocab. Word-based tokenization would require separate models per language.
BPE: The Algorithm That Splits Words
Modern LLMs use Byte Pair Encoding (BPE) to tokenize. The idea is simple but powerful: start with each byte as its own token, then iteratively merge the most common pair into a new token. Repeat until you reach your target vocabulary size (e.g., 100K).
For example, training BPE on English might first merge t+h into th (very common), then th+e into the (extremely common), then th+is into this. After thousands of merges, common words are single tokens; rare words are split into sub-word chunks. The model effectively learns a vocabulary optimized for the data it was trained on.
# Use OpenAI's tokenizer (tiktoken) to see how text gets split
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
text = "The unbelievable hamburger cost $4.99."
tokens = enc.encode(text)
print(f"Text: {text}")
print(f"Token count: {len(tokens)}")
print(f"Tokens: {tokens}")
# Decode each token to see what it represents
for t in tokens:
print(f" {t:6d} -> {repr(enc.decode([t]))}")
Tokenization Matters — A Lot
Here's where it gets practical. Tokenization directly affects 4 things you care about: cost, latency, code quality, and language fairness.
| Impact Area | What This Means | Practical Tip |
|---|---|---|
| Cost | OpenAI charges per token, not per word. 1 English word ≈ 1.3 tokens on average. So 'hello world' = 2 tokens, costs 2× price. | Strip whitespace, avoid repetition, compress prompts |
| Latency | More tokens = slower generation. Each token takes ~30-50ms to generate on GPT-4. | Reduce input tokens; use streaming for perceived speed |
| Code Quality | LLMs struggle with rare identifiers. A function named getQ4Report_v2 might tokenize weirdly. | Use clear, common-word function names when possible |
| Non-English | Chinese characters use 2-3× more tokens than English. Hindi/Korean also inefficient. | Budget for higher cost on non-English pipelines |
Common Tokenization Gotchas
Three gotchas that bite every LLM engineer sooner or later:
- Leading spaces matter:
The(capitalized, start of sentence) andthe(with leading space) are different tokens. This is why your prompt formatting matters — adding or removing spaces can change tokenization and subtly change outputs. - Numbers are weird:
12345might be 1 token, but1234567might be 2-3 tokens. This is why LLMs are bad at arithmetic — the tokenizer splits numbers in unpredictable ways. - Code can balloon tokens: Python with snake_case is token-efficient. JavaScript with camelCase is slightly worse. Minified/obfuscated code can use 5-10× more tokens than readable code.
Practical Exercise
pip install tiktoken🎯 Key Takeaways
- LLMs process tokens, not words or characters.
- BPE (Byte Pair Encoding) splits text into chunks based on training data frequency.
- Common words = 1 token; rare words = multiple tokens; this affects everything downstream.
- Tokenization directly affects cost, latency, code quality, and language fairness.
- Non-English text uses 2-3x more tokens — budget accordingly.
Comments
Comments
Post a Comment