Keyboard Shortcuts N Next post
P Previous post
S Save / unsave
R Read aloud
T Toggle theme
/ Focus search
Esc Close panels
🔥
Ready to read...
LLM Engineering LLM Engineering: From Zero to Production Module 1 - LLM Foundations

Tokenization Explained: How LLMs Actually Read Your Words

Reviewed & accurate
AI Summary

What You Will Learn

BEGINNER

  • What tokens actually are (and why "words" is the wrong mental model)
  • BPE: the algorithm GPT uses to split words into chunks
  • Why tokenization directly affects your cost, latency, and code quality

The Big Misconception: LLMs Don't Read Words

Here's something that surprises every newcomer: LLMs don't process words. They don't process characters either. They process tokens — small chunks of text that might be a whole word, part of a word, a single character, or even punctuation. The model has a fixed vocabulary of tokens (GPT-4's tokenizer knows about 100,000 different tokens), and every input you send is first split into these tokens.

Think of tokens as the "letters" of the LLM's alphabet, except this alphabet has 100,000 symbols instead of 26. The word unbelievable might be split into un + believe + able. The word hamburger might be ham + burger. Or it might be one token. It depends on how often the model saw that exact word during training.

Try It Now
Open OpenAI's Tokenizer tool. Type 'unbelievable' — you'll see it split into 3 tokens. Type 'the' — it's 1 token. Type 'asdfjkl' — it might be 5+ tokens because the model has never seen it. This is why LLMs struggle with rare words, code identifiers, and non-English text.

Why Not Just Words?

You might wonder: why not just split on spaces and call each word a token? Three reasons:

  1. Vocabulary size: English alone has ~1 million words; add names, technical terms, misspellings, and you'd need a vocab of millions. With 100K tokens, the model is manageable.
  2. Sub-word understanding: By splitting unbelievable into un + believe + able, the model learns that un- means "not" and -able means "capable of". This generalizes to words it has never seen (unprogrammable → it can figure out).
  3. Multi-language support: Tokenizers can mix English, Chinese, code, emoji — all in the same vocab. Word-based tokenization would require separate models per language.

BPE: The Algorithm That Splits Words

Modern LLMs use Byte Pair Encoding (BPE) to tokenize. The idea is simple but powerful: start with each byte as its own token, then iteratively merge the most common pair into a new token. Repeat until you reach your target vocabulary size (e.g., 100K).

For example, training BPE on English might first merge t+h into th (very common), then th+e into the (extremely common), then th+is into this. After thousands of merges, common words are single tokens; rare words are split into sub-word chunks. The model effectively learns a vocabulary optimized for the data it was trained on.

tokenize_demo.pypython
# Use OpenAI's tokenizer (tiktoken) to see how text gets split
import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
text = "The unbelievable hamburger cost $4.99."

tokens = enc.encode(text)
print(f"Text: {text}")
print(f"Token count: {len(tokens)}")
print(f"Tokens: {tokens}")

# Decode each token to see what it represents
for t in tokens:
    print(f"  {t:6d}  ->  {repr(enc.decode([t]))}")
Terminal
$ python tokenize_demo.py
Text: The unbelievable hamburger cost $4.99.
Token count: 9
Tokens: [785, 354, 15498, 71389, 2691, 413, 630, 19, 1257, 13]
785 -> 'The'
354 -> ' unbelievable' ← 3 chars? no, it's ' un' + 'believe'...
15498 -> 'believe'
71389 -> 'able'
2691 -> ' hamburger'

Tokenization Matters — A Lot

Here's where it gets practical. Tokenization directly affects 4 things you care about: cost, latency, code quality, and language fairness.

Impact AreaWhat This MeansPractical Tip
CostOpenAI charges per token, not per word. 1 English word ≈ 1.3 tokens on average. So 'hello world' = 2 tokens, costs 2× price.Strip whitespace, avoid repetition, compress prompts
LatencyMore tokens = slower generation. Each token takes ~30-50ms to generate on GPT-4.Reduce input tokens; use streaming for perceived speed
Code QualityLLMs struggle with rare identifiers. A function named getQ4Report_v2 might tokenize weirdly.Use clear, common-word function names when possible
Non-EnglishChinese characters use 2-3× more tokens than English. Hindi/Korean also inefficient.Budget for higher cost on non-English pipelines

Common Tokenization Gotchas

Three gotchas that bite every LLM engineer sooner or later:

  1. Leading spaces matter: The (capitalized, start of sentence) and the (with leading space) are different tokens. This is why your prompt formatting matters — adding or removing spaces can change tokenization and subtly change outputs.
  2. Numbers are weird: 12345 might be 1 token, but 1234567 might be 2-3 tokens. This is why LLMs are bad at arithmetic — the tokenizer splits numbers in unpredictable ways.
  3. Code can balloon tokens: Python with snake_case is token-efficient. JavaScript with camelCase is slightly worse. Minified/obfuscated code can use 5-10× more tokens than readable code.
Token Cost by Language (per 1000 words) English1300 tokens Spanish1500 tokens Chinese2700 tokens Hindi3200 tokens Python1100 tokens JavaScript1400 tokens JSON1700 tokens Minified JS5400 tokens
Approximate token counts per 1000 words/lines. Non-English text costs 2-3x more.

Practical Exercise

Install tiktoken: pip install tiktoken
Run the demo script above. Modify the text to: 'I love Python programming!' and see how it tokenizes.
Now try a Chinese phrase: '你好世界'. Notice it uses more tokens than 'Hello World'. This is why multilingual apps cost more.

🎯 Key Takeaways

  • LLMs process tokens, not words or characters.
  • BPE (Byte Pair Encoding) splits text into chunks based on training data frequency.
  • Common words = 1 token; rare words = multiple tokens; this affects everything downstream.
  • Tokenization directly affects cost, latency, code quality, and language fairness.
  • Non-English text uses 2-3x more tokens — budget accordingly.
Previously: Previously: Lesson 02 covered 70 years of language model history.
Today: Today: you learned how LLMs actually read your words — through tokens, using the BPE algorithm.
Next: Next: Lesson 04 opens the Transformer itself. We'll explain self-attention in plain English, with no scary math.
Test Your Knowledge
How did you find this?

Comments

Join the discussion! Sign in with your Google or Blogger account, or comment as Anonymous - no account needed. For quick questions, also reach me on Telegram @cytestch.

Comments