The Transformer Revolution

Introduction

In June 2017, a team at Google published "Attention Is All You Need" (Vaswani et al.), introducing the Transformer architecture. This paper fundamentally changed the field of deep learning and led directly to models like BERT, GPT, and virtually every modern language model.

In this section, we'll explore the key ideas of the Transformer at a high level, understanding the paradigm shift it represents before diving into implementation details in later chapters.

2.1 The Central Thesis

"Attention Is All You Need"

The paper's title captures its radical proposal:

Remove recurrence entirely. Use only attention mechanisms.

This was counterintuitive at the time. How can you model sequences without any notion of sequential processing?

The answer: Self-attention - a mechanism where every element in a sequence can directly attend to every other element.

2.2 Self-Attention: The Core Innovation

The Key Idea

Instead of processing sequences step-by-step, self-attention computes relationships between ALL pairs of positions simultaneously.

For a sequence of $n$ tokens:

• Each token creates a Query (Q): "What am I looking for?"
• Each token creates a Key (K): "What do I contain?"
• Each token creates a Value (V): "What information do I provide?"

The Attention Formula

Let's unpack this:

1. $QK^T$: Compute similarity between every query and every key ($n \times n$ matrix)
2. $\div \sqrt{d_k}$: Scale to prevent vanishing gradients in softmax
3. softmax: Convert similarities to probabilities (attention weights)
4. $\times V$: Weighted sum of values based on attention weights

Worked Micro-Example (3 tokens, $d_k = 2$)

Visual Intuition

Consider the sentence: "The cat sat on the mat"

1Query from "sat"
2                    ↓
3     The   cat   sat   on   the   mat
4      ↓     ↓     ↓     ↓     ↓     ↓
5Keys: k₁    k₂   k₃    k₄    k₅    k₆
6
7Attention weights: [0.05, 0.35, 0.15, 0.10, 0.05, 0.30]
8                     ↑      ↑                        ↑
9                   "The"  "cat"                    "mat"
10                         (high attention to subject and object)

The word "sat" can directly attend to "cat" (the subject) and "mat" (where it sat), regardless of their positions.

2.3 Why Self-Attention Beats Recurrence

Path Length Comparison

To connect two positions separated by distance $d$:

Parallelization

For a sequence of length $n$:

Self-attention requires $O(n^2)$ operations total, but they can ALL be computed in parallel!

The Computational Trade-off

For modern GPUs optimized for parallel matrix operations:

• $O(n^2)$ parallel operations can be FASTER than $O(n)$ sequential operations
• This is why Transformers train much faster on GPUs than RNNs

2.4 The Transformer Architecture Overview

High-Level Structure

The original Transformer uses an encoder-decoder architecture:

Encoder Stack

Each encoder layer contains:

1. Multi-Head Self-Attention
   • Queries, Keys, Values all come from the same input
   • Each position attends to all positions in the source
2. Feed-Forward Network (FFN)
   • Two linear transformations with ReLU
   • Applied independently to each position
3. Residual Connections + Layer Normalization
   • Around each sub-layer
   • Helps with training deep networks

The encoder layer flow can be summarized as:

Decoder Stack

Each decoder layer contains:

1. Masked Multi-Head Self-Attention
   • Same as encoder, but with causal masking
   • Position $i$ can only attend to positions $\leq i$
   • Prevents "seeing the future" during generation
2. Encoder-Decoder Attention (Cross-Attention)
   • Queries from decoder, Keys/Values from encoder output
   • This is how the decoder "looks at" the source sequence
3. Feed-Forward Network
   • Same as encoder

Encoder/Decoder Data Flow at a Glance

1Input tokens -> Embeddings + Positional Encoding
2              -> Encoder self-attention (Q=K=V=input) -> FFN
3Decoder prefix -> Masked self-attention (Q=K=V=shifted target)
4                -> Cross-attention (Q=decoder, K/V=encoder output)
5                -> FFN -> Linear + Softmax -> Next-token logits
6
7Masks:
8- Padding mask: hide padding in encoder & decoder.
9- Causal mask: hide future tokens in decoder self-attention.

Multi-Head Attention

Instead of single attention, use multiple "heads" in parallel:

where:

Benefits:

• Each head can learn different types of relationships
• Some heads might focus on syntax, others on semantics
• Richer representation of dependencies

2.5 Positional Encoding

The Problem

Self-attention treats input as a set, not a sequence:

• Changing token order doesn't change attention computations
• "cat sat mat" and "mat sat cat" would be identical

The Solution

Add positional encodings to input embeddings:

Sinusoidal Encoding (Original Paper)

Where:

• $pos$ is the position in the sequence
• $i$ is the dimension
• $d_{model}$ is the model dimension

Why Sinusoidal?

1. Bounded values: Between -1 and 1
2. Unique encoding: Each position has a distinct pattern
3. Relative positions: $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$
4. Extrapolation: Works for sequences longer than those seen in training

2.6 Key Architectural Choices

Residual Connections

Every sub-layer has a skip connection:

Benefits:

• Enables training of very deep models
• Gradients can flow directly through skip connections
• Each layer only needs to learn the "residual" (difference from identity)

Layer Normalization

Normalizes across features (not across batch):

Benefits:

• Works with variable-length sequences
• Stable training dynamics
• Works with batch size of 1

The Feed-Forward Network

Dimensions:

• Input/output: $d_{model}$ (e.g., 512)
• Hidden: $d_{ff}$ (e.g., 2048) - typically $4 \times d_{model}$

Role:

• Adds non-linearity
• Processes each position independently
• Acts as a "memory" or lookup table

Encoder Layer: PyTorch-style pseudo-code

1def encoder_layer(x, attn_mask):
2    # x: [batch, seq, d_model]
3    attn_out = multi_head_attention(
4        q=x, k=x, v=x, mask=attn_mask
5    )                          # [batch, seq, d_model]
6    x = layer_norm(x + attn_out)
7
8    ff = relu(x @ W1 + b1)     # W1: [d_model, d_ff]
9    ff = ff @ W2 + b2          # W2: [d_ff, d_model]
10    return layer_norm(x + ff)  # [batch, seq, d_model]

2.7 Training the Transformer

Teacher Forcing

During training:

• Provide the correct previous tokens to decoder
• Don't use model's own predictions (which might be wrong)

Label Smoothing

Instead of hard targets (0 or 1):

• Smooth labels: 0.9 for correct class, $0.1/(V-1)$ for others (where $V$ is vocab size)
• Prevents overconfidence
• Improves generalization

Learning Rate Schedule

The original paper uses a custom schedule:

This creates:

1. Linear warmup: Gradually increase LR
2. Inverse square root decay: Slowly decrease LR

2.8 Why Transformers Succeeded

Superior Performance

Within months of publication:

• Machine Translation: Beat previous SOTA by 2+ BLEU points
• Training Speed: 3.5 days on 8 GPUs (vs. weeks for RNN models)
• Scalability: Performance improved with model size

Hardware Alignment

Transformers are perfectly suited for modern hardware:

• Matrix multiplications ($Q \times K^T$) are GPU-optimized
• No sequential dependencies during training
• Can leverage tensor cores and specialized accelerators

Flexibility

The architecture generalizes to:

• Encoder-only (BERT): Classification, NER, embeddings
• Decoder-only (GPT): Language modeling, generation
• Encoder-decoder (T5, BART): Translation, summarization

Scalability

The same architecture works from:

• Millions of parameters (BERT-base: 110M)
• Billions of parameters (GPT-3: 175B)
• Trillions of parameters (future models)

Mini Timeline

2.9 The Architecture We'll Build

Our Course Project

We'll implement a full encoder-decoder Transformer for German-to-English translation:

1Source (German): "Der Hund ist schwarz."
2                         ↓
3                  [Transformer]
4                         ↓
5Target (English): "The dog is black."

Model Specifications

From Scratch

We'll implement:

1. Scaled dot-product attention
2. Multi-head attention
3. Positional encoding
4. Encoder layer and stack
5. Decoder layer and stack
6. Complete Transformer model

No nn.Transformer wrappers - everything from basic PyTorch operations.

Summary

Key Innovations of the Transformer

1. Self-Attention: Direct connections between any two positions
2. Parallelization: All computations can happen simultaneously
3. Multi-Head Attention: Multiple perspectives on relationships
4. Positional Encoding: Sequence order through learned/fixed patterns

Why It Changed Everything

Glossary / Cheat Sheet

Exercises

Conceptual Questions

1. Why does self-attention have $O(1)$ path length between any two positions, while RNNs have $O(n)$?
2. Explain why Transformers can be parallelized during training while RNNs cannot.
3. What problem does positional encoding solve? What would happen without it?
4. Why do we use multiple attention heads instead of one large attention layer?

Architecture Analysis

1. Draw the flow of information through an encoder layer for processing the sentence "The cat sat".
2. In encoder-decoder attention, where do Q, K, and V come from? Why is this design important?
3. What is the purpose of the feed-forward network in each layer? Why not just use attention?

Hands-on Drills

1. Implement sinusoidal positional encodings in PyTorch or NumPy; verify shapes and plot a few dimensions across positions.
2. Manually compute the output of a 2-head attention layer on three 2D tokens (pick your own tiny numbers) to feel the math.

Looking Ahead

In the next section, we'll survey the incredible variety of applications that Transformers now dominate:

• NLP: Translation, summarization, question answering
• Vision: Image classification, object detection
• Multimodal: CLIP, DALL-E, GPT-4V
• Beyond: Time series, code, robotics

You should now be able to...

1. Explain self-attention and multi-head attention with a concrete numeric example.
2. Sketch the encoder/decoder data flow and point to where masking is required.
3. Recall key hyperparameters ($d_{model}$, heads, $d_{ff}$, warmup) and predict their effect on compute.

Then we'll set up our development environment and preview the German-to-English translation project that will serve as our hands-on practice throughout this course.