Gaussian Distributions Deep Dive

Learning Objectives

By the end of this section, you will:

1. Master the univariate Gaussian PDF and understand every term in the formula
2. Extend to multivariate Gaussians with covariance matrices and understand their geometric interpretation
3. Apply key properties including closure under linear transformations and conditioning
4. Implement the reparameterization trick that enables gradient-based learning through sampling
5. Connect every concept to diffusion models where Gaussians define both forward and reverse processes

The Big Picture

The Gaussian (or Normal) distribution is the most important distribution in all of statistics and machine learning. It was first studied by Abraham de Moivre in 1733 and later developed extensively by Carl Friedrich Gauss, who used it to analyze astronomical data.

Why is the Gaussian so ubiquitous? There are three fundamental reasons:

1. Central Limit Theorem: The sum of many independent random variables converges to a Gaussian, regardless of their individual distributions. Real-world measurements often arise from many small, independent effects—hence the "normal" name.
2. Maximum Entropy: Among all distributions with a given mean and variance, the Gaussian has maximum entropy. It makes the fewest assumptions beyond these constraints, making it the "least informative" choice.
3. Mathematical Convenience: Gaussians are closed under linear operations. Sums of Gaussians are Gaussian. Products of Gaussians are (proportional to) Gaussians. Conditionals and marginals of joint Gaussians are also Gaussian.

The Univariate Gaussian

The Probability Density Function

A random variable X follows a Gaussian distribution with mean $\mu$ and variance $\sigma^2$, written $X \sim \mathcal{N}(\mu, \sigma^2)$, if its probability density function (PDF) is:

$p(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$

Let's understand each component:

Log-Probability

In practice, we almost always work with log-probabilities rather than probabilities to avoid numerical underflow. Taking the log of the Gaussian PDF:

$\log p(x) = -\frac{1}{2}\log(2\pi\sigma^2) - \frac{(x - \mu)^2}{2\sigma^2}$

The log-probability reveals the quadratic structure of the Gaussian. This is why minimizing squared error is equivalent to maximum likelihood estimation with Gaussian noise.

Multivariate Gaussian

Definition

For a random vector $\mathbf{x} \in \mathbb{R}^d$, the multivariate Gaussian with mean vector $\boldsymbol{\mu} \in \mathbb{R}^d$ and covariance matrix $\boldsymbol{\Sigma} \in \mathbb{R}^{d \times d}$ has PDF:

$p(\mathbf{x}) = \frac{1}{(2\pi)^{d/2}|\boldsymbol{\Sigma}|^{1/2}} \exp\left(-\frac{1}{2}(\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x} - \boldsymbol{\mu})\right)$

Each component has a natural interpretation:

Geometric Interpretation

The contours of constant probability density form ellipsoids in d-dimensional space:

• Principal axes are the eigenvectors of $\boldsymbol{\Sigma}$
• Axis lengths are proportional to $\sqrt{\lambda_i}$ where$\lambda_i$ are the eigenvalues
• Spherical when $\boldsymbol{\Sigma} = \sigma^2 I$ (isotropic)
• Axis-aligned ellipse when $\boldsymbol{\Sigma}$ is diagonal
• Rotated ellipse when off-diagonal elements are non-zero

Implementation

mu = [0, 0] for a 2D standard normal

Diagonal Sigma means independent dimensions

z = mu + cholesky(Sigma) @ epsilon

1import torch
2from torch.distributions import MultivariateNormal
3
4# Define parameters
5mean = torch.tensor([0.0, 0.0])
6covariance = torch.tensor([
7    [1.0, 0.5],
8    [0.5, 1.0]
9])
10
11# Create the distribution
12mvn = MultivariateNormal(mean, covariance)
13
14# Sample from the distribution
15samples = mvn.sample((1000,))  # Shape: (1000, 2)
16
17# Compute log-probability
18point = torch.tensor([0.5, 0.5])
19log_prob = mvn.log_prob(point)
20
21# Reparameterization: sample via Cholesky decomposition
22epsilon = torch.randn(2)  # Standard normal
23L = torch.linalg.cholesky(covariance)
24z = mean + L @ epsilon  # Same distribution as mvn.sample()

Key Properties of Gaussians

Property 1: Linear Transformations

If $\mathbf{x} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ and$\mathbf{y} = A\mathbf{x} + \mathbf{b}$, then:

$\mathbf{y} \sim \mathcal{N}(A\boldsymbol{\mu} + \mathbf{b}, A\boldsymbol{\Sigma}A^\top)$

Property 2: Marginals and Conditionals

For a joint Gaussian $\begin{pmatrix} \mathbf{x}_1 \\ \mathbf{x}_2 \end{pmatrix} \sim \mathcal{N}\left(\begin{pmatrix} \boldsymbol{\mu}_1 \\ \boldsymbol{\mu}_2 \end{pmatrix}, \begin{pmatrix} \boldsymbol{\Sigma}_{11} & \boldsymbol{\Sigma}_{12} \\ \boldsymbol{\Sigma}_{21} & \boldsymbol{\Sigma}_{22} \end{pmatrix}\right)$:

• Marginals: $\mathbf{x}_1 \sim \mathcal{N}(\boldsymbol{\mu}_1, \boldsymbol{\Sigma}_{11})$
• Conditionals: $\mathbf{x}_1 | \mathbf{x}_2 \sim \mathcal{N}(\boldsymbol{\mu}_{1|2}, \boldsymbol{\Sigma}_{1|2})$ where:
  • $\boldsymbol{\mu}_{1|2} = \boldsymbol{\mu}_1 + \boldsymbol{\Sigma}_{12}\boldsymbol{\Sigma}_{22}^{-1}(\mathbf{x}_2 - \boldsymbol{\mu}_2)$
  • $\boldsymbol{\Sigma}_{1|2} = \boldsymbol{\Sigma}_{11} - \boldsymbol{\Sigma}_{12}\boldsymbol{\Sigma}_{22}^{-1}\boldsymbol{\Sigma}_{21}$

Property 3: Products of Gaussians

The product of two Gaussian PDFs is proportional to another Gaussian:

$\mathcal{N}(x; \mu_1, \sigma_1^2) \cdot \mathcal{N}(x; \mu_2, \sigma_2^2) \propto \mathcal{N}(x; \mu_*, \sigma_*^2)$

where the precision-weighted combination is:

• $\sigma_*^{-2} = \sigma_1^{-2} + \sigma_2^{-2}$ (precisions add)
• $\mu_* = \sigma_*^2(\sigma_1^{-2}\mu_1 + \sigma_2^{-2}\mu_2)$ (weighted by precision)

Property 4: Sum of Independent Gaussians

If $X \sim \mathcal{N}(\mu_1, \sigma_1^2)$ and$Y \sim \mathcal{N}(\mu_2, \sigma_2^2)$ are independent:

$X + Y \sim \mathcal{N}(\mu_1 + \mu_2, \sigma_1^2 + \sigma_2^2)$

The Reparameterization Trick

One of the most important techniques in modern generative modeling is thereparameterization trick. The problem it solves: how do we backpropagate gradients through random sampling?

The Problem

Suppose we want to optimize parameters $\theta$ of a distribution$p_\theta(z)$ with respect to some loss $\mathcal{L}(z)$. We need:

$\nabla_\theta \mathbb{E}_{z \sim p_\theta}[\mathcal{L}(z)]$

But sampling $z \sim p_\theta$ is a discrete, non-differentiable operation! The gradient cannot flow through the sampling step.

The Solution

For Gaussians, we can reparameterize the sampling:

1. Instead of sampling $z \sim \mathcal{N}(\mu, \sigma^2)$ directly...
2. Sample $\epsilon \sim \mathcal{N}(0, 1)$ (independent of parameters)
3. Compute $z = \mu + \sigma \cdot \epsilon$ (deterministic given $\epsilon$)

Now the expectation becomes:

$\mathbb{E}_{z \sim p_\theta}[\mathcal{L}(z)] = \mathbb{E}_{\epsilon \sim \mathcal{N}(0,1)}[\mathcal{L}(\mu + \sigma \cdot \epsilon)]$

And we can differentiate through $z = \mu + \sigma \cdot \epsilon$!

dL/d(mu) = dL/dz * dz/d(mu) = dL/dz * 1

1import torch
2from torch.distributions import Normal
3
4# Parameters from a neural network (require gradients)
5mu = torch.tensor([1.0], requires_grad=True)
6log_sigma = torch.tensor([0.0], requires_grad=True)  # Use log for positivity
7sigma = torch.exp(log_sigma)
8
9# Sample noise from standard normal
10epsilon = torch.randn_like(mu)
11
12# Reparameterize: z = mu + sigma * epsilon
13z = mu + sigma * epsilon
14
15# Compute some loss and backpropagate
16loss = (z ** 2).sum()  # Example loss
17loss.backward()
18
19# Gradients now flow through mu and sigma!
20print(f"Gradient w.r.t. mu: {mu.grad}")
21print(f"Gradient w.r.t. log_sigma: {log_sigma.grad}")

Bonus: The Central Limit Theorem

The CLT explains why Gaussians appear everywhere. If $X_1, X_2, \ldots, X_n$ are independent and identically distributed (i.i.d.) with mean $\mu$ and variance $\sigma^2$, then as $n \to \infty$:

$\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} \mathcal{N}(0, 1)$

Gaussians in Diffusion Models

Let's connect everything to diffusion models explicitly:

Why This Matters

1. Forward process is fully determined: We can compute$q(x_t|x_0)$ in closed form for any t, enabling efficient training.
2. Reverse posterior is tractable: Given $x_0$, we know the exact $q(x_{t-1}|x_t, x_0)$. This becomes our training target.
3. KL divergences have closed forms: Between two Gaussians, the KL divergence is analytic, enabling efficient ELBO computation.
4. Reparameterization enables training: We sample$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon$and train to predict $\epsilon$.

Summary

In this section, we developed a deep understanding of Gaussian distributions:

1. Univariate Gaussian: The PDF has a normalizing constant and exponential of quadratic deviation from the mean. Log-probability reveals the connection to squared error loss.
2. Multivariate Gaussian: Extends to vectors with mean vectors and covariance matrices. Contours form ellipsoids whose axes are determined by eigenvectors.
3. Key Properties: Closure under linear transformations, tractable marginals and conditionals, products combine via precision weighting, sums of independent Gaussians are Gaussian.
4. Reparameterization Trick: Enables gradient-based learning by separating randomness ($\epsilon$) from parameters ($\mu, \sigma$).
5. Diffusion Connection: Both forward and reverse processes are defined by Gaussians, making the entire framework analytically tractable.

Exercises

Conceptual Questions

1. Explain why the Gaussian distribution maximizes entropy among all distributions with fixed mean and variance. What does this imply for modeling uncertainty?
2. Given a 2D Gaussian with correlation $\rho = 0.9$, describe qualitatively what the contours look like. What happens as$\rho \to 1$?
3. Why does the reparameterization trick work for Gaussians but not for all distributions (e.g., discrete distributions)?
4. In the diffusion forward process, why do we use$\sqrt{\bar{\alpha}_t}$ and$\sqrt{1-\bar{\alpha}_t}$ as coefficients rather than arbitrary functions?

Computational Exercises

1. Implement a function that computes the KL divergence between two univariate Gaussians $\mathcal{N}(\mu_1, \sigma_1^2)$ and$\mathcal{N}(\mu_2, \sigma_2^2)$ using the closed-form formula. Verify empirically using Monte Carlo sampling.
2. Given a bivariate Gaussian with $\mu = [0, 0]$ and$\Sigma = \begin{bmatrix} 1 & 0.7 \\ 0.7 & 1 \end{bmatrix}$, compute the conditional distribution $p(x_1 | x_2 = 1.5)$. Sample from both the joint and the conditional to verify.
3. Implement the reparameterization trick for a VAE encoder that outputs$\mu$ and $\log\sigma$. Verify that gradients flow correctly by computing $\partial L/\partial \mu$for a simple loss.
4. Simulate the diffusion forward process for a 1D distribution. Start with samples from a mixture of Gaussians, apply the forward process for T=100 steps with a linear noise schedule, and visualize how the distribution evolves toward$\mathcal{N}(0, 1)$.

Challenge Problem

Derive the posterior $q(x_{t-1}|x_t, x_0)$ for the diffusion process:

1. Start with $q(x_t|x_{t-1})$ and$q(x_{t-1}|x_0)$
2. Use Bayes' theorem: $q(x_{t-1}|x_t, x_0) \propto q(x_t|x_{t-1})q(x_{t-1}|x_0)$
3. Show that the product of two Gaussians gives a Gaussian with specific mean and variance
4. Verify your result matches the formula in the DDPM paper