Forging Giants: Training Massive Models from Scratch

Part I · Chapter 01 · 6 sections · 130 min

The specific mathematical tools that recur throughout large-scale training: tensors, SVD, probability, autodiff, optimisation, and numerical precision.

1. 1.1Tensors, Matrix Multiply, and Linear Maps20m
2. 1.2Eigendecomposition, SVD, and Low-Rank Approximation25m
3. 1.3Probability, Entropy, and Cross-Entropy Loss20m
4. 1.4Automatic Differentiation and the Chain Rule20m
5. 1.5Gradient Descent, Adam, and Optimisation Landscapes25m
6. 1.6Numerical Precision Fundamentals20m

Part I · Chapter 02 · 7 sections · 150 min

Build the complete transformer architecture by deriving each component from the problem it solves.

1. 2.1The Sequence Modelling Problem15m
2. 2.2Scaled Dot-Product Attention: Derivation25m
3. 2.3Multi-Head Attention20m
4. 2.4Positional Encodings and RoPE25m
5. 2.5Feed-Forward Networks as Memory20m
6. 2.6Layer Normalisation and Training Stability20m
7. 2.7Full Transformer Forward Pass: End-to-End with Shapes25m

Part I · Chapter 03 · 6 sections · 85 min

The full tokenisation pipeline from raw text to token IDs, and the design decisions that affect model quality.

1. 3.1Why Subword Tokenisation?10m
2. 3.2Byte-Pair Encoding from Scratch20m
3. 3.3SentencePiece and Unigram LM Tokenisation15m
4. 3.4Vocabulary Size Trade-offs10m
5. 3.5Special Tokens and Chat Templates15m
6. 3.6Tokenisation Pitfalls15m

Part II · Chapter 04 · 6 sections · 130 min

The KV cache bottleneck at scale and MLA's low-rank compression solution derived completely from scratch.

1. 4.1The KV Cache Bottleneck20m
2. 4.2Grouped-Query and Multi-Query Attention20m
3. 4.3MLA: Low-Rank Joint Compression — Full Derivation30m
4. 4.4Decoupled RoPE20m
5. 4.5MLA vs GQA vs MHA: Full Comparison15m
6. 4.6Implementing MLA in PyTorch25m

Part II · Chapter 05 · 6 sections · 125 min

The MoE architecture from sparse conditional compute, DeepSeekMoE fine-grained expert decomposition, and expert parallelism.

1. 5.1Why Mixture-of-Experts?15m
2. 5.2The Routing Mechanism25m
3. 5.3Fine-Grained Expert Decomposition20m
4. 5.4Shared Experts15m
5. 5.5Expert Parallelism at Scale25m
6. 5.6Implementing DeepSeekMoE25m

Part II · Chapter 06 · 4 sections · 75 min

Why MoE load imbalance is catastrophic, why auxiliary loss hurts quality, and how DeepSeek's bias-term solution resolves the tension.

1. 6.1The Routing Collapse Problem15m
2. 6.2Auxiliary Loss Approaches and Their Cost20m
3. 6.3Bias-Term Load Balancing: The DeepSeek Solution25m
4. 6.4Sequence-Level Balance Loss15m

Part II · Chapter 07 · 5 sections · 95 min

Why next-token prediction under-uses the forward pass, DeepSeek's sequential MTP, and speculative decoding at inference.

1. 7.1The Single-Token Prediction Bottleneck15m
2. 7.2Naive Parallel MTP and Its Failure Mode15m
3. 7.3Sequential Causal MTP: DeepSeek's Implementation25m
4. 7.4MTP Training Objective and Ablations20m
5. 7.5MTP as Speculative Decoding at Inference20m

Part III · Chapter 08 · 7 sections · 120 min

Build a complete data pipeline capable of producing 14.8T high-quality training tokens.

1. 8.1The Data Pipeline Architecture20m
2. 8.2Deduplication at Scale20m
3. 8.3Quality Filtering20m
4. 8.4Data Mixing and Domain Weighting15m
5. 8.5Synthetic Data: When and How15m
6. 8.6Training Data Curriculum15m
7. 8.7Data Contamination Detection15m

Part III · Chapter 09 · 6 sections · 120 min

Determine the right model size and training token count for a given compute budget using theory and evidence.

1. 9.1The Chinchilla Scaling Laws25m
2. 9.2MoE Scaling Laws20m
3. 9.3Emergent Abilities15m
4. 9.4Hyperparameter Scaling20m
5. 9.5Predicting Final Loss from Intermediate Checkpoints15m
6. 9.6Inference-Aware Scaling Laws25m

Part III · Chapter 10 · 6 sections · 120 min

Why FP8 training is hard, DeepSeek's fine-grained quantisation and high-precision accumulation solutions.

1. 10.1The Case for FP815m
2. 10.2Why Naive FP8 Training Fails20m
3. 10.3Fine-Grained Quantisation25m
4. 10.4High-Precision Accumulation20m
5. 10.5What Stays in BF16 and FP3215m
6. 10.6Implementing FP8 Training25m

Part III · Chapter 11 · 8 sections · 160 min

All four parallelism strategies and DeepSeek's DualPipe algorithm that eliminates communication bottlenecks.

1. 11.1Why One GPU Is Not Enough10m
2. 11.2Data Parallelism (DP)20m
3. 11.3Tensor Parallelism (TP)20m
4. 11.4Pipeline Parallelism and the Bubble Problem25m
5. 11.5DualPipe: DeepSeek's Solution30m
6. 11.6Expert Parallelism and Cross-Node All-to-All20m
7. 11.7Memory Optimisation: No Tensor Parallelism Required20m
8. 11.8Checkpoint Strategy and Fault Tolerance15m

Part III · Chapter 12 · 4 sections · 65 min

Why extending context is non-trivial, and the YaRN technique used by DeepSeek V3 to reach 128K tokens.

1. 12.1Why Context Extension Is Hard15m
2. 12.2NTK-Aware Scaling15m
3. 12.3YaRN: Frequency-Domain Interpolation25m
4. 12.4Evaluation at Long Context10m

Part IV · Chapter 13 · 5 sections · 80 min

SFT as a formatting and style adapter on top of a knowledge-rich base model.

1. 13.1What SFT Actually Does15m
2. 13.2SFT Data Collection15m
3. 13.3Chat Template and Formatting15m
4. 13.4SFT Training Configuration20m
5. 13.5Catastrophic Forgetting and Mitigation15m

Part IV · Chapter 14 · 7 sections · 140 min

The full RLHF pipeline from preference data to a trained reward model and PPO training.

1. 14.1The Alignment Problem10m
2. 14.2The Bradley-Terry Model for Preferences20m
3. 14.3Reward Model Architecture and Training20m
4. 14.4Rule-Based Rewards for Verifiable Tasks15m
5. 14.5Generative Reward Models (LLM-as-Judge)15m
6. 14.6PPO: The Standard RLHF Algorithm30m
7. 14.7DPO: Direct Preference Optimization30m

Part IV · Chapter 15 · 6 sections · 130 min

Derive GRPO from PPO, eliminate the critic, and implement GRPO from scratch with all reward shaping choices.

1. 15.1The Critic Bottleneck in PPO15m
2. 15.2GRPO Derivation30m
3. 15.3GRPO Hyperparameters from DeepSeek R115m
4. 15.4Reward Design for Reasoning20m
5. 15.5GRPO Variants: DAPO, Dr.GRPO, and Olmo 320m
6. 15.6Implementing GRPO from Scratch30m

Part IV · Chapter 16 · 6 sections · 80 min

The R1-Zero experiment — its hypothesis, results, emergent phenomena, and what it reveals about LLM reasoning.

1. 16.1The R1-Zero Hypothesis15m
2. 16.2Experimental Setup10m
3. 16.3The Aha Moment15m
4. 16.4Quantitative Results15m
5. 16.5Failure Modes of R1-Zero10m
6. 16.6What R1-Zero Teaches Us15m

Part IV · Chapter 17 · 7 sections · 105 min

The multi-stage pipeline from base model to production reasoning model, including distillation to smaller models.

1. 17.1Pipeline Overview15m
2. 17.2Cold-Start SFT: Why a Few Thousand Examples Help15m
3. 17.3Stage 2: Reasoning-Focused GRPO15m
4. 17.4Stage 3: Rejection Sampling for SFT Data15m
5. 17.5Stage 4: Alignment GRPO15m
6. 17.6Distillation to Smaller Models20m
7. 17.7Distilling Reasoning into DeepSeek V3 Chat10m

Part V · Chapter 18 · 5 sections · 100 min

Prefill/decode split, KV cache management, speculative decoding, quantisation, and expert load balancing.

1. 18.1Two Very Different Problems: Prefill vs Decode20m
2. 18.2KV Cache Management and PagedAttention25m
3. 18.3Speculative Decoding with MTP20m
4. 18.4Post-Training Quantisation20m
5. 18.5Expert Load Balancing at Inference15m

Part V · Chapter 19 · 5 sections · 95 min

Design and operate a serving system for a 671B MoE model at production scale.

1. 19.1The DeepSeek Production Architecture20m
2. 19.2Inference Framework Comparison15m
3. 19.3Continuous Batching and Request Scheduling20m
4. 19.4Multi-Node Inference and Network Requirements20m
5. 19.5Cost Modelling and Optimisation20m

Part V · Chapter 20 · 5 sections · 90 min

Rigorous evaluation practices, production monitoring, and the future of massive model training.

1. 20.1Benchmark Taxonomy15m
2. 20.2Evaluation Contamination Detection15m
3. 20.3Production Monitoring20m
4. 20.4The Future: What DeepSeek's Work Reveals15m
5. 20.5Reproducing DeepSeek on a Budget: A Practical Roadmap25m