Course Roadmap and Project Preview

1YOUR TRANSFORMER JOURNEY
2════════════════════════════════════════════════════════════════════════
3
4  START                                                              FINISH
5    │                                                                   │
6    ▼                                                                   ▼
7┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────────┐
8│  "What  │    │ "I see  │    │"I built │    │"My model│    │"I understand│
9│   is    │───▶│  how    │───▶│  the    │───▶│ learns! │───▶│ deeply and  │
10│attention│    │ Q,K,V   │    │encoder- │    │BLEU: 30+│    │ can modify/ │
11│    ?"   │    │  work"  │    │decoder" │    │         │    │   extend"   │
12└─────────┘    └─────────┘    └─────────┘    └─────────┘    └─────────────┘
13     │              │              │              │               │
14  Ch 1-2         Ch 3-4         Ch 7-8        Ch 10-14        Ch 15-17
15  ~1 week        ~1 week        ~1 week       ~2 weeks        Optional
16
17════════════════════════════════════════════════════════════════════════
18              Total: 4-8 weeks depending on your background

1Chapter 1-4:   Foundations     (Attention, Multi-Head, Position, Embeddings)
2                     ↓
3Chapter 5-6:   Building Blocks (Tokenization, FFN, Normalization)
4                     ↓
5Chapter 7-9:   Core Architecture (Encoder, Decoder, Generation)
6                     ↓
7Chapter 10-11: Training & Evaluation (Pipeline, BLEU)
8                     ↓
9Chapter 12-14: Complete Project (Data, Training, Analysis)
10                     ↓
11Chapter 15-17: Advanced Topics (Variants, Modern Methods, Production)

1TIME COMMITMENT OVERVIEW
2────────────────────────────────────────────────────────────────────
3Part 1: Foundations      ████████████████░░░░░░░░░░░░░░░  ~2 weeks
4Part 2: Building Blocks  ████████░░░░░░░░░░░░░░░░░░░░░░░  ~1 week
5Part 3: Core Arch        ████████░░░░░░░░░░░░░░░░░░░░░░░  ~1 week
6Part 4: Training/Eval    ████████░░░░░░░░░░░░░░░░░░░░░░░  ~1 week
7Part 5: Full Project     ████████████████░░░░░░░░░░░░░░░  ~2 weeks
8Part 6: Advanced         ████████████░░░░░░░░░░░░░░░░░░░  Optional
9────────────────────────────────────────────────────────────────────
10                         │         │         │         │
11                         5hrs      10hrs     15hrs     20hrs/week

1# What we DO:
2def scaled_dot_product_attention(Q, K, V, mask=None):
3    d_k = Q.size(-1)
4    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
5    if mask is not None:
6        scores = scores.masked_fill(mask == 0, -1e9)
7    attention_weights = F.softmax(scores, dim=-1)
8    return torch.matmul(attention_weights, V), attention_weights
9
10# What we DON'T do (until comparison at the end):
11output = nn.MultiheadAttention(embed_dim, num_heads)(query, key, value)

1# What you'll implement:
2- Dot-product similarity
3- Softmax attention weights
4- Weighted value aggregation
5- Scaling factor derivation
6- Masking mechanisms

1# What you'll implement:
2class MultiHeadAttention(nn.Module):
3    def __init__(self, d_model, num_heads):
4        # Projection matrices
5        self.W_Q = nn.Linear(d_model, d_model)
6        self.W_K = nn.Linear(d_model, d_model)
7        self.W_V = nn.Linear(d_model, d_model)
8        self.W_O = nn.Linear(d_model, d_model)

1# What you'll implement:
2- Sinusoidal positional encoding
3- Learned positional embeddings
4- Token embedding layer
5- Combined input representations

1# What you'll implement:
2class BPETokenizer:
3    def train(self, corpus, vocab_size):
4        # Learn merge rules from data
5
6    def encode(self, text):
7        # Convert text to token IDs
8
9    def decode(self, token_ids):
10        # Convert token IDs back to text

1# What you'll implement:
2class PositionwiseFeedForward(nn.Module):
3    def __init__(self, d_model, d_ff, dropout=0.1):
4        self.linear1 = nn.Linear(d_model, d_ff)
5        self.linear2 = nn.Linear(d_ff, d_model)
6        self.dropout = nn.Dropout(dropout)
7
8class LayerNorm(nn.Module):
9    # From scratch, understanding why it helps training

1# What you'll implement:
2class EncoderLayer(nn.Module):
3    # Self-attention + FFN + residuals + normalization
4
5class TransformerEncoder(nn.Module):
6    # Stack of N encoder layers

1Input embeddings + positions
2         ↓
3┌────────────────────────┐
4│    Self-Attention      │←─┐
5├────────────────────────┤  │ × N layers
6│ Add & Norm             │──┘
7├────────────────────────┤
8│    Feed-Forward        │←─┐
9├────────────────────────┤  │
10│ Add & Norm             │──┘
11└────────────────────────┘
12         ↓
13    Encoder output

1# What you'll implement:
2class DecoderLayer(nn.Module):
3    def forward(self, x, encoder_output, src_mask, tgt_mask):
4        # Masked self-attention
5        x = self.self_attention(x, x, x, tgt_mask)
6        # Cross-attention to encoder
7        x = self.cross_attention(x, encoder_output, encoder_output, src_mask)
8        # Feed-forward
9        x = self.feed_forward(x)
10        return x

1# What you'll implement:
2def greedy_decode(model, encoder_output, max_len):
3    """Generate one token at a time, feeding back predictions."""
4
5def beam_search(model, encoder_output, beam_width, max_len):
6    """Maintain multiple hypotheses for better results."""

1# What you'll implement:
2class TransformerTrainer:
3    def train_epoch(self, dataloader):
4        for batch in dataloader:
5            loss = self.compute_loss(batch)
6            loss.backward()
7            self.optimizer.step()
8            self.scheduler.step()

1# What you'll implement:
2def compute_bleu(references, hypotheses, max_n=4):
3    """Compute BLEU score from scratch."""
4
5def corpus_bleu(references, hypotheses):
6    """Corpus-level BLEU with brevity penalty."""

1# ~30,000 sentence pairs
2# Example:
3german:  "Ein Mann in einem blauen Hemd steht auf einer Leiter."
4english: "A man in a blue shirt is standing on a ladder."

1class Transformer(nn.Module):
2    def __init__(self, config):
3        self.encoder = TransformerEncoder(config)
4        self.decoder = TransformerDecoder(config)
5        self.src_embed = Embeddings(config)
6        self.tgt_embed = Embeddings(config)
7        self.generator = nn.Linear(config.d_model, config.tgt_vocab_size)

1DE: Eine Gruppe von Menschen steht vor einem Iglu.
2EN: A group of people stands in front of an igloo.
3
4DE: Ein kleines Mädchen klettert in ein Spielhaus aus Holz.
5EN: A little girl climbing into a wooden playhouse.
6
7DE: Ein Mann in einem blauen Hemd steht auf einer Leiter.
8EN: A man in a blue shirt is standing on a ladder.

1Epoch 1:   5-10  (learning basic patterns)
2Epoch 5:   15-20 (capturing word relationships)
3Epoch 10:  25-30 (good fluency, some errors)
4Epoch 20:  30-35 (our target)
5Epoch 30+: 35-40 (diminishing returns)

1┌─────────────────────────────────────────────────────────────────────┐
2│  YOUR MODEL'S LEARNING JOURNEY                                      │
3├─────────────────────────────────────────────────────────────────────┤
4│                                                                     │
5│  Input: "Ein Mann liest ein Buch"                                   │
6│                                                                     │
7│  Epoch 1:  "A the the the the"        ❌ Gibberish                  │
8│            └─ Model just repeating common tokens                    │
9│                                                                     │
10│  Epoch 5:  "A man is a book"          ⚠️ Words present, wrong order │
11│            └─ Learning word mappings, not structure                 │
12│                                                                     │
13│  Epoch 10: "A man reading a book"     ⚠️ Close! Wrong verb form     │
14│            └─ Getting syntax, some errors                           │
15│                                                                     │
16│  Epoch 15: "A man reads a book"       ✅ Correct!                   │
17│            └─ Learned the pattern                                   │
18│                                                                     │
19│  Epoch 20: "A man in a blue shirt     ✅ Handles complexity!        │
20│             reads a book in the park"                               │
21│            └─ Generalizes to longer sentences                       │
22│                                                                     │
23└─────────────────────────────────────────────────────────────────────┘

1# 🎯 YOUR FIRST WIN: See attention patterns in 60 seconds
2# Copy this into a Python file or Jupyter notebook
3
4import torch
5import torch.nn.functional as F
6import matplotlib.pyplot as plt
7
8# Simulate a mini attention computation
9sentence = ["The", "cat", "sat", "on", "the", "mat"]
10seq_len = len(sentence)
11
12# Random Q, K, V matrices (normally these come from embeddings)
13d_k = 8
14Q = torch.randn(seq_len, d_k)
15K = torch.randn(seq_len, d_k)
16V = torch.randn(seq_len, d_k)
17
18# THE ATTENTION FORMULA (you'll implement this properly in Chapter 2)
19scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
20attention_weights = F.softmax(scores, dim=-1)
21
22# Visualize: which words attend to which?
23plt.figure(figsize=(8, 6))
24plt.imshow(attention_weights.detach().numpy(), cmap='Blues')
25plt.xticks(range(seq_len), sentence, rotation=45)
26plt.yticks(range(seq_len), sentence)
27plt.xlabel('Keys (attending TO)')
28plt.ylabel('Queries (attending FROM)')
29plt.title('Your First Attention Map!')
30plt.colorbar(label='Attention Weight')
31plt.tight_layout()
32plt.savefig('my_first_attention.png')
33plt.show()
34
35print("✅ You just computed attention from scratch!")
36print("📊 Saved: my_first_attention.png")
37print("🎯 Next: Chapter 2 will explain WHY this formula works")

1PREREQUISITE CHECKLIST
2─────────────────────────────────────────────────
3□ I can write a Python class with __init__ and methods     [  /5]
4□ I can reshape a NumPy array from (10, 20) to (20, 10)    [  /5]
5□ I can define a simple nn.Module in PyTorch               [  /5]
6□ I know what A @ B means for matrices                     [  /5]
7□ I understand train/val/test splits                       [  /5]
8─────────────────────────────────────────────────
9                                          Total: [  /25]
10
11Score 20+: You're ready! Dive in.
12Score 15-19: You'll be fine, expect some Googling.
13Score <15: Spend a day on PyTorch basics first.

1Python >= 3.8
2PyTorch >= 2.0
3torchtext (for data utilities)
4matplotlib (for visualization)
5tqdm (for progress bars)

1CUDA-capable GPU (training is 10-20x faster)
2Jupyter/Colab (for interactive exploration)
3wandb (for experiment tracking)

1# Create a fresh environment
2conda create -n transformers python=3.10
3conda activate transformers
4
5# Install PyTorch (check pytorch.org for your CUDA version)
6pip install torch torchvision torchaudio
7
8# Install course dependencies
9pip install torchtext matplotlib tqdm numpy
10
11# Verify installation
12python -c "import torch; print(f'PyTorch {torch.__version__}, CUDA: {torch.cuda.is_available()}')"
13
14# Expected output: PyTorch 2.x.x, CUDA: True (or False if no GPU)

1transformer_translation/
2├── data/
3│   ├── multi30k/
4│   │   ├── train.de
5│   │   ├── train.en
6│   │   ├── val.de
7│   │   ├── val.en
8│   │   ├── test.de
9│   │   └── test.en
10│   └── vocab/
11│       ├── bpe.de
12│       └── bpe.en
13├── models/
14│   ├── attention.py
15│   ├── encoder.py
16│   ├── decoder.py
17│   ├── transformer.py
18│   └── embeddings.py
19├── training/
20│   ├── trainer.py
21│   ├── scheduler.py
22│   └── losses.py
23├── evaluation/
24│   ├── bleu.py
25│   └── analysis.py
26├── utils/
27│   ├── tokenizer.py
28│   ├── dataset.py
29│   └── visualization.py
30├── configs/
31│   └── base_config.yaml
32├── checkpoints/
33├── notebooks/
34│   └── exploration.ipynb
35└── train.py

1# Add this to EVERY function you write:
2def my_attention(Q, K, V):
3    print(f"Q: {Q.shape}, K: {K.shape}, V: {V.shape}")  # Always check inputs
4
5    # ... your code ...
6
7    print(f"output: {output.shape}")  # Always check outputs
8    return output
9
10# Expected shapes to memorize:
11# Q, K, V: (batch, heads, seq_len, d_k)
12# Attention weights: (batch, heads, seq_len, seq_len)
13# Output: (batch, heads, seq_len, d_k)

1# Pitfall 1: Tensor shape confusion
2# ALWAYS check shapes with print statements
3print(f"Q shape: {Q.shape}")  # Expected: (batch, heads, seq_len, d_k)
4
5# Pitfall 2: Forgetting to handle padding
6# Masks are essential for variable-length sequences
7
8# Pitfall 3: Not scaling attention
9# The 1/√d_k factor prevents gradient vanishing
10
11# Pitfall 4: Wrong mask dimensions
12# src_mask: (batch, 1, 1, src_len)
13# tgt_mask: (batch, 1, tgt_len, tgt_len)
14
15# Pitfall 5: Not handling start/end tokens
16# <sos> and <eos> tokens are critical for generation

1AFTER THIS COURSE, YOU CAN CONFIDENTLY SAY:
2─────────────────────────────────────────────────────────────────────
3✅ "I implemented a transformer from scratch in PyTorch"
4
5✅ "I understand why we scale attention by √d_k—it's to prevent
6    softmax saturation when dimensions are large"
7
8✅ "I built a German-English translator that achieved BLEU 32,
9    competitive with baseline models"
10
11✅ "I can explain the difference between encoder self-attention,
12    decoder self-attention, and cross-attention"
13
14✅ "I've debugged attention masks—the batch and head dimensions
15    need to broadcast correctly"
16
17✅ "I know when to use encoder-only (BERT) vs decoder-only (GPT)
18    vs encoder-decoder (T5) architectures"
19─────────────────────────────────────────────────────────────────────

1YOUR ACHIEVEMENT CHECKLIST
2════════════════════════════════════════════════════════════════════
3□ UNDERSTAND transformers deeply, not just use them as black boxes
4□ IMPLEMENT every component from scratch in PyTorch
5□ BUILD a working German-to-English translation system
6□ ACHIEVE a BLEU score of 30-35, competitive with baselines
7□ VISUALIZE and interpret attention mechanisms
8□ DEBUG transformer issues confidently (shapes, masks, gradients)
9□ EXPLAIN the architecture in job interviews
10□ APPLY these skills to other transformer-based projects
11════════════════════════════════════════════════════════════════════

1import torch
2import torch.nn as nn
3
4class TestModule(nn.Module):
5    def __init__(self):
6        super().__init__()
7        self.linear = nn.Linear(10, 5)
8
9    def forward(self, x):
10        return self.linear(x)
11
12# Test it
13model = TestModule()
14x = torch.randn(2, 10)
15print(model(x).shape)  # Should print: torch.Size([2, 5])
16print("✅ Environment ready! You're all set for Chapter 2.")

1def attention(Q, K, V):
2    """
3    The heart of transformers.
4
5    Q: What am I looking for? (Query)
6    K: What do I contain? (Key)
7    V: What information do I provide? (Value)
8
9    Returns: Weighted combination of Values
10    """
11    scores = Q @ K.T / sqrt(d_k)
12    weights = softmax(scores)
13    return weights @ V