Transformer Implementation in PyTorch

Part I · Chapter 00 · 3 sections · 45 min

Essential knowledge before diving into Transformers

1. 0.1Prerequisite Knowledge10m
2. 0.2The Core Insight: How Transformers Work15m
3. 0.3Softmax and Cross-Entropy Loss20m

Part I · Chapter 01 · 4 sections · 45 min

The evolution of sequence modeling and the Transformer revolution

1. 1.1Evolution of Sequence Modeling12m
2. 1.2The Transformer Revolution15m
3. 1.3Real World Applications and Variants10m
4. 1.4Course Roadmap and Project Preview8m

Part I · Chapter 02 · 6 sections · 92 min

Understanding and implementing the core attention mechanism

1. 2.1Intuition Behind Attention12m
2. 2.2Scaled Dot Product Attention Formula15m
3. 2.3Numerical Walkthrough18m
4. 2.4Implementing Attention in PyTorch20m
5. 2.5Understanding and Implementing Masks15m
6. 2.6Attention Visualization and Debugging12m

Part I · Chapter 03 · 5 sections · 69 min

Parallel attention heads for richer representations

1. 3.1Why Multiple Heads10m
2. 3.2Linear Projections for QKV12m
3. 3.3Reshaping for Parallel Heads15m
4. 3.4Implementing MultiHeadAttention20m
5. 3.5Self vs Cross Attention12m

Part I · Chapter 04 · 6 sections · 87 min

Adding position information to the Transformer

1. 4.1The Position Problem10m
2. 4.2Sinusoidal Positional Encoding15m
3. 4.3Learned Positional Embeddings12m
4. 4.4Token Embeddings10m
5. 4.5Combined Embedding Layer15m
6. 4.6Modern Positional Encodings25m

Part II · Chapter 05 · 5 sections · 70 min

Building vocabulary with subword tokenization

1. 5.1Why Subword Tokenization10m
2. 5.2Byte Pair Encoding Algorithm15m
3. 5.3Implementing BPE from Scratch20m
4. 5.4Using SentencePiece for Production15m
5. 5.5Special Tokens for Seq2Seq10m

Part III · Chapter 06 · 4 sections · 49 min

The building blocks that complete each layer

1. 6.1Position-wise Feed Forward Networks12m
2. 6.2Layer Normalization Deep Dive15m
3. 6.3Residual Connections10m
4. 6.4The Add and Norm Pattern12m

Part III · Chapter 07 · 5 sections · 67 min

Building the complete encoder stack

1. 7.1Encoder Layer Architecture12m
2. 7.2Implementing TransformerEncoderLayer18m
3. 7.3Stacking Encoder Layers12m
4. 7.4Complete Encoder Forward Pass15m
5. 7.5Source Sentence Encoding10m

Part III · Chapter 08 · 6 sections · 95 min

Building the decoder with masked attention

1. 8.1Decoder Architecture Overview12m
2. 8.2Causal Masking15m
3. 8.3Cross Attention15m
4. 8.4Implementing TransformerDecoderLayer18m
5. 8.5Complete Transformer Decoder15m
6. 8.6Full Encoder-Decoder Transformer20m

Part IV · Chapter 09 · 5 sections · 75 min

Generating sequences token by token

1. 9.1Understanding Autoregressive Generation12m
2. 9.2Greedy Decoding15m
3. 9.3Beam Search18m
4. 9.4Sampling Strategies15m
5. 9.5KV Caching15m

Part V · Chapter 10 · 5 sections · 74 min

Complete training setup for translation

1. 10.1Data Loading and Batching15m
2. 10.2Label Smoothing Loss12m
3. 10.3Learning Rate Scheduling15m
4. 10.4Complete Training Loop20m
5. 10.5Checkpointing and Model Selection12m

Part V · Chapter 11 · 4 sections · 52 min

Measuring translation quality

1. 11.1Introduction to Translation Metrics10m
2. 11.2BLEU Score Implementation15m
3. 11.3Other Evaluation Metrics12m
4. 11.4Practical Evaluation Pipeline15m

Part VI · Chapter 12 · 4 sections · 55 min

Preparing the translation dataset

1. 12.1Multi30k Dataset Overview10m
2. 12.2Data Preprocessing15m
3. 12.3Vocabulary and Tokenizer Setup15m
4. 12.4Data Loading Pipeline15m

Part VI · Chapter 13 · 3 sections · 50 min

Training our Transformer on German-English translation

1. 13.1Model Configuration and Setup15m
2. 13.2Complete Training Script20m
3. 13.3Training Monitoring and Debugging15m

Part VI · Chapter 14 · 2 sections · 27 min

Using the trained model for translation

1. 14.1Inference Pipeline15m
2. 14.2Interactive Demo and Conclusion12m

Part VII · Chapter 15 · 3 sections · 45 min

Leveraging pretrained models for translation

1. 15.1Introduction to Pretrained Models12m
2. 15.2Finetuning mBART18m
3. 15.3Advanced Finetuning Techniques15m

Part VII · Chapter 16 · 3 sections · 48 min

Modern improvements to the Transformer

1. 16.1Flash Attention15m
2. 16.2Mixture of Experts18m
3. 16.3Modern Position Encodings15m

Part VII · Chapter 17 · 2 sections · 30 min

Deploying Transformers in production

1. 17.1Model Optimization15m
2. 17.2Model Export and Serving15m