Attention Is All You Need: The Paper, Explained
A beginner-friendly guide to the Transformer — the architecture behind ChatGPT, Gemini, Claude, and almost every modern AI model. Every technical term is defined. Every concept is grounded in analogy.
The Big Picture
In 2017, the best way to make a computer translate a sentence — or do almost anything with language — was to feed it the words one at a time, in order, using a kind of network that kept a running memory of what it had seen so far. These were called recurrent neural networksA recurrent neural network (RNN) processes a sequence one element at a time, updating an internal "memory" (hidden state) at each step and passing it forward. Powerful for sequences, but inherently sequential.. They worked, but they had a stubborn flaw: because each word had to wait for the previous word to be processed, you couldn't do the work in parallel — and modern hardware (GPUsGraphics Processing Units — chips with thousands of small cores that do many calculations at once. They make deep learning fast, but only if the work can be split into parallel pieces. RNNs couldn't.) is built to do thousands of things at once.
The Transformer's radical proposal was in its title: attention is all you need. Throw out the step-by-step recurrence entirely. Instead, let every word look directly at every other word in the sentence, all at the same time, and decide for itself which other words matter. That single idea — called self-attentionA mechanism that lets each word in a sentence directly "look at" and pull information from every other word, computing for itself how relevant each one is. The core building block of the Transformer. — is what the whole paper is built on. The problems it solved:
- Recurrence forbids parallelism. Processing word 10 required finishing words 1 through 9 first. On a chip designed to run thousands of operations simultaneously, that's like owning a highway and being forced to drive single file. The Transformer processes all words at once.
- Long-range connections were fragile. To link the first word of a paragraph to the last, an RNN had to pass information through every word in between — a long, lossy game of telephone. The Transformer connects any two words directly, in a single step, no matter how far apart.
- Training was slow and expensive. Because of the above, top translation models took a long time and a lot of compute. The Transformer beat them while training in a small fraction of the time.
See It In Action
Before any math, here's the intuition you should hold onto. When a model reads a sentence, the word “it” is ambiguous — what does it refer to? Self-attention lets the word “it” look at every other word and decide. Click any word below to see (illustratively) which other words it would pay the most attention to. Notice that “it” leans heavily on “animal”.
That's the entire promise of attention, in miniature: relevance is computed, not assumed. The model isn't told that “it” refers to “animal” — it learns to compute a high relevance score between them. Now let's build up how that actually works.
Background Concepts
This paper assumes you're comfortable with the basics of neural networks for language. Let's build that vocabulary from scratch — each idea here is a prerequisite for understanding the Transformer.
From words to numbers: tokens and embeddings
A neural network only understands numbers, not letters. So the first step in any language model is to chop text into pieces called tokensThe small units a model reads — often words or word-fragments. "unbelievable" might become "un", "believ", "able". The paper uses ~32,000–37,000 sub-word tokens via byte-pair encoding. (roughly, words or word-fragments), then map each token to a list of numbers called an embeddingA vector (a list of numbers, e.g. 512 of them) that represents a token's meaning. Words with similar meanings get similar vectors. The model learns these during training.. An embedding is just a vectorAn ordered list of numbers, like [0.2, -1.1, 0.7, ...]. You can think of it as coordinates pointing to a location in a high-dimensional "meaning space." — a point in a high-dimensional “meaning space” where similar words sit near each other. In this paper every token becomes a vector of 512 numbers.
The old workhorse: recurrent networks and their bottleneck
Before the Transformer, sequence problems were ruled by RNNsRecurrent Neural Networks — networks that read a sequence step by step, carrying a memory (hidden state) forward from one step to the next. and their smarter cousins, LSTMsLong Short-Term Memory networks — a type of RNN with gates that help it remember information over longer stretches without it fading away. and GRUsGated Recurrent Units — a streamlined LSTM variant. Like LSTMs, designed to carry memory across many steps.. They read a sentence the way you read aloud: one word at a time, updating a mental summary (the hidden stateThe RNN's running memory — a vector updated at each step that's supposed to summarize everything seen so far.) as you go. This is intuitive, but it forces two costs: the work is strictly sequential (word 10 can't start until word 9 is done), and information from far-back words has to survive a long relay to reach the present, often fading along the way.
Encoder and decoder: read, then write
Translation models share a common shape: an encoderThe half of the model that reads and "understands" the input sentence, turning it into a set of rich numeric representations. reads the whole input sentence and compresses its meaning into a set of vectors; a decoderThe half that writes the output sentence, one token at a time, using both what it has written so far and the encoder's representation of the input. then writes the output sentence using that understanding. The decoder works autoregressively"Auto-regressive" means it generates one token at a time, and each new token is fed back in as input to help predict the next one. This is how GPT-style models still generate text today. — it produces one word, feeds that word back in, produces the next, and so on. The Transformer keeps this encoder–decoder shape but rebuilds both halves out of attention.
Softmax: turning scores into a spotlight
One small function shows up everywhere in this paper: softmaxA function that turns a list of raw scores into positive numbers that add up to 1 (a probability distribution). Big scores get exaggerated; small ones get squashed toward zero.. It takes a list of raw numbers (“scores”) and turns them into percentages that add up to 100%, exaggerating the big ones and shrinking the small ones. Attention uses softmax to turn raw “how relevant is this word?” scores into clean weights — a spotlight that mostly lands on the few words that matter and dims the rest.
The Heart of It: Attention
Everything in the Transformer rests on one operation. The paper describes attention with three roles that every word plays at once: a Query, a Key, and a Value. The names come from databases, and the analogy is the clearest way in.
What I'm looking for. Each word forms a question: “which other words are relevant to me?”
What I advertise about myself. Every word offers a label that queries can match against.
What I actually contribute. The information a word hands over if it gets attended to.
Scaled dot-product attention
Here is the actual recipe, in plain steps. For one query word:
- Score every word. Compare the query against each word's key using a dot productA way of multiplying two vectors into a single number that measures how aligned they are. Big when the vectors point the same way — a natural "similarity" score. — a single number measuring how well they match.
- Scale the scores. Divide every score by √dkThe square root of the key dimension (d_k = 64 in the base model). Dividing by it keeps the scores from growing too large, which would push softmax into a region where it barely learns. A small but crucial stabilizer. to keep them from getting too large.
- Softmax into weights. Run the scores through softmax so they become positive weights that sum to 1 — the attention spotlight.
- Blend the values. Take a weighted sum of all the words' values, using those weights. The result is the query word's new, context-aware representation.
The paper writes this as one compact formula, Attention(Q, K, V) = softmax(QKT / √dk) V, and computes it for all words simultaneously as a single matrix multiplication — which is exactly why it's so fast on a GPU. The video below walks through it for the word “it.”
Why divide by √dk? (the “scaled” part)
When you take a dot product of two long vectors of random-ish numbers, the result tends to grow larger as the vectors get longer (specifically, its variance grows with the dimension dkThe dimension of the key and query vectors — 64 in the base model (512 total, split across 8 heads).). Very large scores push softmax into a regime where almost all the weight piles onto one option and the gradientsThe signals used to train a network — they tell each weight which direction to adjust. If softmax saturates, these signals shrink toward zero and learning stalls. become tiny, so the model learns slowly. Dividing by √dk rescales the scores back to a sane range, keeping training stable. It's a one-character fix with an outsized effect.
Self-attention vs. cross-attention
The same operation gets used in two modes. In self-attention, the queries, keys, and values all come from the same sentence — words looking at their own neighbors to build context (this is what the “it” → “animal” demo showed). In cross-attention, the queries come from the sentence being written (the decoder) while the keys and values come from the input sentence (the encoder) — this is how the translation looks back at the original. Same math, different sources.
Many heads are better than one
A single attention operation can only focus one way at a time. But a word relates to others in many ways at once — grammatically, semantically, by position. So the Transformer runs attention multiple times in parallel, each with its own learned projections, and calls each one a headOne independent attention computation. The base Transformer uses 8 heads, each operating on a 64-dimensional slice, so the total cost matches a single full-size attention.. The paper uses 8 heads. Each head can specialize — one might track the subject of a verb, another the next word, another a long-range reference — and their outputs are concatenated and mixed back together.
The Transformer, Assembled
Now we can stack the pieces into the full model. The Transformer keeps the classic two-tower shape — an encoder that reads the input and a decoder that writes the output — but each tower is a stack of 6 identical layers built from attention and simple feed-forward networks.
A few components in that diagram deserve a closer look.
The feed-forward network: thinking about each word on its own
After attention has mixed information between words, each word passes through a small feed-forward networkA simple two-layer transformation applied to each position independently: expand to 2048 dimensions, apply a ReLU non-linearity, shrink back to 512. It lets the model process the information attention just gathered. — the same little two-step transformation applied to every position separately. If attention is the step where words talk to each other, the feed-forward network is the step where each word thinks privately about what it just heard. It expands each 512-number vector to 2048, applies a ReLURectified Linear Unit — a simple non-linear function that keeps positive numbers and zeroes out negatives. It lets the network learn non-linear, more expressive patterns. non-linearity, and shrinks it back.
Residual connections and layer normalization: keeping deep stacks trainable
Around every sub-layer, the Transformer adds two pieces of plumbing. A residual connectionA shortcut that adds a sub-layer's input to its output (output = x + Sublayer(x)). It gives gradients a clear path through deep networks, making them much easier to train. adds each sub-layer's input back to its output, giving information (and training signal) a clean shortcut through all 6 layers. Then layer normalizationA step that re-centers and re-scales a vector to have a consistent statistical range. It stabilizes and speeds up training of deep networks. rescales the result to a stable range. These aren't glamorous, but without them, stacking 6+ layers of attention would be far harder to train.
Positional encoding: putting the words back in order
Here's a subtle problem. Because the Transformer looks at all words at once instead of in sequence, it has no inherent idea of word order — to it, “dog bites man” and “man bites dog” would look identical. The fix is positional encodingA pattern of numbers added to each word's embedding that encodes its position in the sentence. The paper uses sine and cosine waves of different frequencies so every position gets a unique, learnable-to-interpret signature.: before the first layer, each word's embedding gets a unique “position fingerprint” added to it, built from sine and cosine waves of different frequencies.
Why sine waves instead of just numbering the positions 1, 2, 3…?
Plain position numbers don't generalize well and grow without bound. The sinusoidal scheme has two nice properties. First, every position gets a distinct pattern across the 512 dimensions (slow waves capture coarse position, fast waves capture fine position). Second — and this is the clever part — for any fixed offset k, the encoding of position pos+k is a simple linear function of the encoding at pos, which the authors hypothesized would make it easy for the model to learn to attend by relative position ("the word three back"). They also tried learned position vectors and got nearly identical results — but chose sinusoids because they might extrapolate to sentences longer than any seen in training.
Masking: why the decoder can't peek ahead
When the decoder writes word number 5, it must not be allowed to see words 6, 7, 8… — those don't exist yet at generation time, and letting the model see them during training would be cheating. So the decoder's self-attention is maskedA trick that blocks attention to future positions by setting their scores to negative infinity before softmax (which makes their weights zero). It preserves the auto-regressive property: position i can only use positions up to i.: any attention weight pointing to a future word is forced to zero. We'll see this in motion in the inference section.
Why Self-Attention Wins
Section 4 of the paper makes the efficiency argument explicit by comparing self-attention to recurrent and convolutional layers on three measures. The headline is about path lengthThe number of steps information must travel to get from one position to another through the network. Shorter paths make long-range dependencies easier to learn.: how many steps must information travel to connect two words?
| Layer type | Compute per layer | Sequential steps | Max path length |
|---|---|---|---|
| Self-Attention | O(n² · d) | O(1) | O(1) |
| Recurrent (RNN/LSTM) | O(n · d²) | O(n) | O(n) |
| Convolutional | O(k · n · d²) | O(1) | O(logk n) |
Here n is the number of words and d is the vector size. The two columns that matter: self-attention needs only O(1) sequential steps (everything happens in one parallel pass) and has an O(1) maximum path length (any word reaches any other directly). Recurrent layers need O(n) on both — the bottleneck the Transformer was designed to escape.
The catch: that O(n²) compute term
Self-attention isn't free. Because every word attends to every other word, the compute grows with the square of the sentence length (n²). For the sentence lengths in this paper that's a great trade — it's cheaper than recurrence whenever n is smaller than the vector size d, which is usually true. But that same n² term is exactly why, years later, a whole research field sprang up to make attention cheaper for very long inputs (think book-length documents). The authors even floated restricting attention to a local neighborhood as future work. The Transformer won by spending compute to buy parallelism and short paths — a bet that paid off as hardware kept getting faster.
How It's Trained
The Transformer was trained on standard machine-translation datasets — about 4.5 million English–German sentence pairs, and a larger English–French set — with a few important tricks that made training fast and stable.
The learning-rate warmup schedule
The learning rateHow big a step the optimizer takes when adjusting the model's weights. Too high and training diverges; too low and it crawls. isn't constant. The paper increases it linearly for the first 4,000 warmup stepsAn initial phase where the learning rate ramps up from near zero. It prevents the unstable early updates that a large fixed rate would cause in a fresh, randomly-initialized model., then decreases it in proportion to the inverse square root of the step number. The intuition: a brand-new model is fragile, so take small steps at first; once it finds its footing, take bigger steps, then gradually settle down. This warmup trick became standard practice for training Transformers. They used the Adam optimizerA popular optimization algorithm that adapts the step size for each weight automatically. The workhorse optimizer for deep learning. to do the actual weight updates.
Dropout and label smoothing
DropoutDuring training, randomly switch off a fraction of the network's connections each step. This stops the model from over-relying on any one path and reduces overfitting. randomly disables 10% of connections during training so the model can't lean too hard on any single pathway — a classic defense against overfittingWhen a model memorizes its training data instead of learning general patterns, so it performs well in practice runs but poorly on new examples.. Label smoothingInstead of training the model to be 100% certain of the correct word, train it toward, say, 90% — leaving a little probability spread across others. It hurts raw confidence but improves accuracy and translation quality. stops the model from becoming overconfident: rather than demanding it predict the correct word with 100% certainty, it's trained toward a slightly softer target. Counter-intuitively this worsens the model's confidence metric but improves its actual accuracy and translation scores.
The whole base model trained in about 12 hours on 8 GPUs; the larger model took 3.5 days — a small fraction of what competing models of the day required.
How It Generates Text
Once trained, the Transformer generates a translation the same way GPT-style models still generate text today: one token at a time, feeding each new token back in to predict the next. At every step, the decoder attends to the full input sentence (cross-attention) and to the words it has already written — but, thanks to masking, never to words it hasn't written yet.
Beam search: not just grabbing the single best word each step
Greedily picking the highest-probability word at every step can paint you into a corner — an early choice that looks good locally might doom the rest of the sentence. So at inference the paper uses beam searchA search strategy that keeps the few most promising partial sentences ("beams") alive at each step instead of committing to one, then picks the best complete sentence at the end. The paper uses a beam size of 4., which keeps the top 4 candidate sentences alive simultaneously and only commits at the end. A small length penaltyAn adjustment that keeps beam search from unfairly preferring very short outputs (which tend to have higher total probability just by being shorter). keeps it from preferring suspiciously short translations.
Results
The Transformer didn't just match the previous best translation systems — it beat them, while training far more cheaply. Translation quality is measured in BLEUBilingual Evaluation Understudy — a score from 0 to 100 measuring how closely a machine translation matches human reference translations. Higher is better; a couple of points is a meaningful gap., where higher is better.
| Model | EN→DE BLEU | EN→FR BLEU | Training cost (FLOPs) |
|---|---|---|---|
| GNMT + RL (Google, 2016) | 24.6 | 39.9 | ~1.4 × 1020 |
| ConvS2S (Facebook) | 25.2 | 40.5 | ~1.5 × 1020 |
| Previous best ensemble | 26.4 | 41.3 | ~1.2 × 1021 |
| Transformer (base) | 27.3 | 38.1 | 3.3 × 1018 |
| Transformer (big) | 28.4 | 41.8 | 2.3 × 1019 |
Two things stand out. The big Transformer set a new state of the art on English→German (28.4 BLEU, more than 2 points above the previous best, including ensembles) and English→French (41.8). And the cost column is the real shock: even the base model trained for 10–100× less compute — measured in FLOPsFloating-point operations — the total count of arithmetic operations a computer performs. A standard way to measure how much raw computation training a model takes., the total arithmetic operations — than the competition. More quality, far less cost.
What the ablations taught us
The authors also varied the model piece by piece to see what mattered. The lessons: too few attention heads hurt quality, but so did too many (8 was a sweet spot); shrinking the key size hurt, suggesting matching words is genuinely hard; bigger models were reliably better; and dropout was essential. They also confirmed the Transformer generalizes beyond translation by applying it successfully to English sentence parsingConstituency parsing — analyzing the grammatical structure of a sentence into a tree. A very different task from translation, used here to show the architecture is general., a structurally very different task.
Final Quiz
Why This Paper Matters
For builders and practitioners
If you've used ChatGPT, Claude, Gemini, Copilot, or virtually any modern AI product, you've used a descendant of this exact architecture. The Transformer turned out to be far more than a translation model: because it's parallelizable and scales gracefully, it became the substrate for large language modelsModels like GPT, Claude, and Gemini, trained on vast amounts of text. They are, at their core, very large Transformers., image models, speech models, and protein-folding models. The practical lessons are still copied verbatim today — multi-head attention, residual-plus-normalization blocks, warmup schedules, sub-word tokenization. Learning this paper isn't studying history; it's learning the blueprint your tools are built from.
For the research community
The paper's deepest contribution was a removal, not an addition: it showed that the recurrence everyone assumed was essential for sequence modeling could be dropped entirely. That cleared the way for scaling. Within a year, BERT and GPT (the first big Transformer-based language models) showed the same architecture, trained on raw text at scale, could be repurposed for almost any language task — kicking off the pretraining era. The Transformer also made models more interpretable (you can inspect attention) and more general (the same block works across modalities). Few papers have ever changed a field's default architecture this completely.
The bigger picture
“Attention Is All You Need” is arguably the most consequential machine-learning paper of its decade. It didn't just win a translation benchmark; it handed the field an architecture that kept getting better as you made it bigger and fed it more data — the property that, scaled up a thousandfold, produced the generative-AI moment we're living through now. The authors closed by saying they were “excited about the future of attention-based models” and planned to extend them beyond text to images, audio, and video. That turned out to be a dramatic understatement. The entire modern AI landscape is, in a real sense, a footnote to this 2017 paper.
Glossary
The neural-network architecture introduced by this paper, built entirely from attention instead of recurrence. The foundation of modern large AI models.
An operation where each element scores how relevant every other element is, then blends their information by those scores.
Attention where queries, keys, and values all come from the same sequence — words building context from their own neighbors.
Attention where queries come from the decoder and keys/values from the encoder — how the output looks back at the input.
The three roles each word plays: what it's looking for, what it advertises, and what it contributes when matched.
Running several attention computations in parallel, each on a slice of the vector, so different heads capture different relationships. The paper uses 8.
The specific attention recipe: dot-product scores, divided by √d_k, softmaxed into weights, used to blend values.
The stack that reads the input sentence and turns it into context-rich vectors.
The stack that writes the output one token at a time, using its own past output and the encoder's vectors.
Recurrent networks that process sequences step by step, carrying a memory forward. What the Transformer replaced.
The small unit of text a model reads — a word or word-fragment.
A vector of numbers representing a token's meaning; similar tokens get similar vectors.
Turns raw scores into positive weights that sum to 1, exaggerating big scores and shrinking small ones.
Multiplying two vectors into one number that measures how aligned they are — a similarity score.
A small per-word transformation (expand to 2048, ReLU, shrink to 512) applied after attention.
A shortcut adding a sub-layer's input to its output, easing gradient flow through deep stacks.
Re-scales a vector to a stable statistical range, stabilizing training.
Position "fingerprints" (sine/cosine waves) added to embeddings so the model knows word order.
Blocking attention to future positions (scores set to −∞) so the decoder can't peek ahead.
Generating one token at a time, feeding each output back in to predict the next.
Keeping the few most promising partial outputs alive during generation, then choosing the best complete one.
How many steps information travels to connect two positions. O(1) for self-attention, O(n) for RNNs.
A 0–100 score for how closely a machine translation matches human references. Higher is better.
Ramping the learning rate up for the first 4,000 steps, then decaying it — key to stable Transformer training.
Training toward slightly-less-than-100% confidence in the correct token; hurts confidence but improves accuracy.
Randomly switching off connections during training to prevent overfitting.