Keyboard Shortcuts N Next post
P Previous post
S Save / unsave
R Read aloud
T Toggle theme
/ Focus search
Esc Close panels
🔥
Ready to read...
LLM Engineering LLM Engineering: From Zero to Production Module 1 - LLM Foundations

Inside the Transformer: Attention Explained Like You're 10

Reviewed & accurate
AI Summary

What You Will Learn

INTERMEDIATE

  • Self-attention in one sentence (then unpacked properly)
  • Q, K, V matrices without the math terror
  • Multi-head attention: why "more heads = better"

Self-Attention in One Sentence

Self-attention is a mechanism where each word in a sentence decides how much to "pay attention" to every other word, so that the meaning of each word is enriched by the context around it. That's it. Everything else — Q/K/V matrices, softmax scaling, multi-head — is just an efficient way to compute this attention.

Let's make this concrete. Take the sentence: "The bank of the river was muddy." The word bank is ambiguous — it could mean a financial institution or the side of a river. How does the model know which meaning to use? Because bank "attends to" river and muddy, which together push its meaning toward the river-side sense. Without attention, the model would have to guess based on the word alone. With attention, the meaning of every word is computed dynamically from the surrounding words.

Self-Attention: Each Word Looks at Every Other Word The bank of the river was muddy attends → context → 'bank' learns it's a river bank, not a financial bank
Word 'bank' attends to 'river' and 'muddy' — and now knows it means the river-side sense.

The Q, K, V Story (Without the Math Terror)

Now the technical version. Each token in your input has three things computed from it: a Query (Q), a Key (K), and a Value (V). All three are vectors — lists of numbers — produced by multiplying the token's embedding by three learned weight matrices (also called Q, K, V).

Here's the intuition that makes it click:

  • Query (Q): "What am I looking for?" — When the word bank wants to figure out its meaning, its Query is something like "I'm ambiguous; I need context about bodies of water or money."
  • Key (K): "What do I have?" — Each other word has a Key that advertises what it offers. river's Key might say "I'm about water, nature, geography."
  • Value (V): "Here's what I'll give you." — When bank and river have a high Q-K match, river sends its Value (its semantic content) to enrich bank's representation.

The match between a Query and a Key is just a dot product (cosine similarity, basically). High dot product = strong attention = lots of value flowing from that word.

The Library Analogy
Imagine you walk into a library. Your Query is 'I want books about river ecosystems.' Each book's Key is its title and table of contents. You compare your query against every book's key — books with matching keywords get high scores. Then you read the Value (the actual content) of the books that matched, ignoring the rest. Self-attention does this for every word, simultaneously.

The Math (Just Enough, I Promise)

The full attention formula is shorter than you think:

attention.pypython
# Self-attention in 4 lines of NumPy
import numpy as np

def attention(Q, K, V):
    # Q, K, V are matrices of shape (seq_len, d_k)
    scores = Q @ K.T / np.sqrt(K.shape[-1])  # how much each Q matches each K
    weights = softmax(scores, axis=-1)       # normalize to probabilities
    return weights @ V                        # weighted sum of values

The 1/sqrt(d_k) scaling is just to keep the dot products from getting too large (which would make softmax saturate and gradients vanish). Softmax turns the raw scores into a probability distribution (sums to 1). The final multiply gives you a weighted blend of all the Values — this is the new, context-enriched representation of the current word.

Multi-Head Attention: Why More Heads = Better

A single attention mechanism can only learn one type of relationship at a time. But language has many: subject-verb agreement, adjective-noun modification, negation scope, coreference (which "he" or "she" refers to), long-range dependencies, etc. The Transformer solves this with multi-head attention: instead of one Q/K/V, you have N (typically 8, 16, or 96) parallel "heads", each with its own learned Q/K/V matrices.

Each head learns a different relationship. Head 1 might focus on subject-verb agreement. Head 2 on adjective-noun. Head 3 on long-range dependencies. And so on. The outputs of all heads are concatenated and passed through a final linear layer to mix them. This is why models with more heads generally do better — they can model more relationships simultaneously.

Model# HeadsHead DimTotal Attn Dim
GPT-2 small1264768
GPT-3 (175B)9612812,288
Llama 3 8B321284,096
Llama 3 70B641288,192
GPT-4 (est.)12012815,360

Why Attention Is Fast (and Slow)

Self-attention is O(n²) in sequence length — every word attends to every other word, so a 100K-token context means 10 billion attention scores to compute. This is why long-context models (1M+ tokens) require clever tricks like sparse attention, sliding window attention, or context compression.

The flip side: attention is fully parallelizable across positions (unlike RNNs which must process sequentially). This is why Transformers train much faster on GPUs than RNNs did — every token's attention can be computed in parallel.

🌍 REAL WORLD: KV Cache — The Trick That Makes Generation Fast
When you generate tokens one at a time, you'd waste time recomputing K and V for all previous tokens every step. The fix: cache the K and V vectors of all previous tokens. Now each new token only needs to compute its own Q, and dot-product it with the cached K's. This is called the KV cache, and it's why ChatGPT can stream responses at 50+ tokens/second. Tools like vLLM (we'll cover it in Lesson 64) make this dramatically more efficient with PagedAttention.

Practical Exercise

Read the original paper's Section 3.2.1 on attention: arxiv.org/abs/1706.03762. It's only 1 page.
Try the excellent visualizer at Transformer Explainer — see attention weights light up in real time.
Think about this: if attention is O(n²), what happens to memory if you double the context length? (Answer: 4× memory. This is why long-context models are expensive.)

🎯 Key Takeaways

  • Self-attention lets every word look at every other word to gather context.
  • Q (query), K (key), V (value) — like searching a library: query says what you want, key advertises what's there, value is the content.
  • Multi-head attention learns multiple relationships in parallel — more heads = richer understanding.
  • Attention is O(n²) in sequence length — this is why long context is expensive.
  • KV caching makes generation fast by not recomputing old K and V vectors.
Previously: Previously: Lesson 03 showed how text becomes tokens.
Today: Today: you went inside the Transformer and learned how self-attention works — the engine of every modern LLM.
Next: Next: Lesson 05 covers the training pipeline: pre-training, fine-tuning, and RLHF.
Test Your Knowledge
How did you find this?

Comments

Join the discussion! Sign in with your Google or Blogger account, or comment as Anonymous - no account needed. For quick questions, also reach me on Telegram @cytestch.

Comments