Historical Context and Key Papers

Learning Objectives

By the end of this section, you will be able to:

1. Trace the evolution of diffusion models from thermodynamics to modern AI
2. Identify the key papers that established the theoretical and practical foundations
3. Understand the connections between score matching, denoising, and diffusion
4. Appreciate the rapid progress from research to commercial applications
5. Recognize emerging trends and future research directions

The Diffusion Timeline

The story of diffusion models spans decades, from physics to modern deep learning. Understanding this history helps appreciate why the field developed as it did.

Foundational Work (2015-2019)

Sohl-Dickstein et al. (2015): Deep Unsupervised Learning

The first paper to propose diffusion probabilistic models for generative modeling. The key insight: model generation as the reverse of a gradual noising process.

Key Innovation: "We develop a deep generative model that learns to generate samples by reversing a diffusion process. The model is trained by maximizing a variational lower bound on the data log-likelihood."

While theoretically beautiful, this paper didn't achieve state-of-the-art results at the time. The models were slow to sample and didn't match GANs in quality. But the seed was planted.

Song & Ermon (2019): Score-Based Generative Models (NCSN)

This paper approached generation from a different angle: learning the score function (gradient of log-density) rather than the density itself.

Key insights:

• Denoising score matching: Train by predicting noise added to clean data
• Annealed Langevin dynamics: Sample by following the learned score with decreasing noise levels
• Multi-scale training: Learn scores at multiple noise levels for better results

This work established the crucial connection between denoising and score estimation, which would later unify with the DDPM framework.

The DDPM Breakthrough (2020)

Ho et al. (2020): Denoising Diffusion Probabilistic Models

This paper changed everything. By making several key simplifications, the authors showed that diffusion models could match or exceed GANs in image quality.

The DDPM Insight: "We show that diffusion models are capable of generating high quality samples. We achieve this by proposing a weighted variational bound that emphasizes certain terms in the ELBO."

Key Simplifications in DDPM

1. Fixed variance: Set$\Sigma_\theta = \beta_t I$ instead of learning it
2. Noise prediction: Parameterize the network to predict$\epsilon$ rather than$\mu$
3. Simple loss: Use unweighted MSE:$L = \|\epsilon - \epsilon_\theta(x_t, t)\|^2$
4. Uniform timestep sampling: Sample$t$ uniformly during training

The training objective became embarrassingly simple:

Song et al. (2021): Score-Based Generative Modeling through SDEs

This paper unified the score-matching and diffusion perspectives through stochastic differential equations (SDEs).

Key contributions:

• Unified framework: DDPM, NCSN, and continuous diffusion are all special cases
• ODE formulation: Probability flow ODE enables deterministic sampling
• Exact likelihood: Can compute exact log-likelihood via the ODE formulation

Rapid Progress (2021-2022)

DDIM (Song et al., 2021): Faster Sampling

A key limitation of DDPM was slow sampling (1000 steps). DDIM introduced a non-Markovian formulation that enables fewer steps:

• 10-50x faster sampling with minimal quality loss
• Deterministic sampling option (useful for interpolation)
• Same trained model, different inference procedure

Classifier Guidance (Dhariwal & Nichol, 2021)

This paper showed how to guide sampling with a classifier, dramatically improving conditional generation:

The paper also introduced ADM (Ablated Diffusion Model), achieving state-of-the-art image quality on ImageNet.

Classifier-Free Guidance (Ho & Salimans, 2022)

An even simpler approach: train unconditionally and conditionally in the same model, then interpolate at inference:

This became the standard for text-to-image models, enabling the quality-diversity trade-off via the guidance scale $w$.

Latent Diffusion (Rombach et al., 2022)

A crucial efficiency improvement: run diffusion in a compressed latent space rather than pixel space.

• Autoencoder compression: 4-8x spatial compression
• Faster training and inference: Much smaller images
• Stable Diffusion: Open-source release revolutionized the field

The Modern Era (2023+)

Major Applications

Efficiency Improvements

• Progressive Distillation: Reduce steps through student-teacher training
• Consistency Models: Direct mapping from noise to data in one step
• LCM (Latent Consistency Models): Fast high-quality generation in 4-8 steps
• SDXL Turbo: Real-time image generation

Architecture Innovations

• DiT (Diffusion Transformers): Replace U-Net with Transformer architecture
• Rectified Flows: Straighter sampling paths for faster inference
• Flow Matching: Simpler training objectives with similar performance

Key Papers to Read

For a deep understanding of diffusion models, these papers are essential:

Foundational Theory

1. Sohl-Dickstein et al. (2015): "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" - The original paper
2. Song & Ermon (2019): "Generative Modeling by Estimating Gradients of the Data Distribution" - Score matching
3. Ho et al. (2020): "Denoising Diffusion Probabilistic Models" - The DDPM breakthrough
4. Song et al. (2021): "Score-Based Generative Modeling through Stochastic Differential Equations" - Unified framework

Practical Improvements

1. Song et al. (2021): "Denoising Diffusion Implicit Models" (DDIM) - Faster sampling
2. Dhariwal & Nichol (2021): "Diffusion Models Beat GANs on Image Synthesis" - Classifier guidance
3. Ho & Salimans (2022): "Classifier-Free Diffusion Guidance" - Simpler conditioning
4. Rombach et al. (2022): "High-Resolution Image Synthesis with Latent Diffusion Models" - Stable Diffusion

Applications

1. Ramesh et al. (2022): "Hierarchical Text-Conditional Image Generation with CLIP Latents" - DALL-E 2
2. Saharia et al. (2022): "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding" - Imagen
3. Poole et al. (2022): "DreamFusion: Text-to-3D using 2D Diffusion" - 3D generation

Future Directions

The field is evolving rapidly. Key research directions include:

Efficiency

• One-step or few-step generation (consistency models, rectified flows)
• Better distillation techniques
• Hardware-aware architectures

Scalability

• Larger models with better scaling laws
• More efficient training procedures
• Better pretraining and fine-tuning paradigms

New Modalities

• Long-form video generation
• 3D scene generation
• Multimodal generation (unified models)
• Scientific applications (molecules, materials, proteins)

Control and Editing

• More precise spatial and semantic control
• Better composition and reasoning
• Consistent character/style generation

Summary

The history of diffusion models spans from physics to AI:

1. Origins (2015): Sohl-Dickstein connected thermodynamic diffusion to generative modeling
2. Score Matching (2019): Song & Ermon showed denoising learns score functions
3. DDPM (2020): Ho et al. simplified training to make diffusion practical
4. Unification (2021): Song et al. connected everything through SDEs
5. Applications (2022+): Text-to-image, video, 3D revolutionized by diffusion

Looking Ahead: Part I is now complete! You have the mathematical prerequisites (Gaussians, information theory, variational inference, Markov chains) and the conceptual foundation (generative modeling, model landscape, diffusion intuition, history). In Part II, we'll dive into the formal mathematics of the forward and reverse diffusion processes.