Information Theory Essentials

Learning Objectives

By the end of this section, you will:

1. Understand entropy as a measure of uncertainty and information content
2. Master cross-entropy and its connection to classification loss functions
3. Apply KL divergence to measure the difference between probability distributions
4. Recognize the asymmetry of KL divergence and when to use forward vs reverse KL
5. Connect information theory to diffusion models through the ELBO and training objectives

The Big Picture

Information theory was founded by Claude Shannon in 1948 with his landmark paper "A Mathematical Theory of Communication." Shannon asked a fundamental question:How can we quantify information?

Shannon's key insight: information is about reducing uncertainty. The more uncertain we are about an outcome, the more information we gain when we observe it. This leads to a natural mathematical definition of information and entropy.

In machine learning, information theory provides the foundation for:

Shannon Entropy

Definition and Intuition

The entropy of a discrete random variable X with distribution p(x) is:

$H(X) = -\sum_{x} p(x) \log p(x) = \mathbb{E}\left[-\log p(X)\right]$

Each term $-\log p(x)$ is called the surprisal orself-information of outcome x. Entropy is the expected surprisal.

Properties of Entropy

1. Non-negativity: $H(X) \geq 0$. Entropy is zero only when X is deterministic.
2. Maximum entropy: For a discrete variable over K outcomes, entropy is maximized by the uniform distribution: $H_{\max} = \log K$.
3. Additivity for independence: If X and Y are independent,$H(X, Y) = H(X) + H(Y)$.
4. Conditioning reduces entropy: $H(X|Y) \leq H(X)$. Knowledge of Y can only reduce (or maintain) uncertainty about X.

Continuous Entropy (Differential Entropy)

For continuous random variables with PDF p(x):

$h(X) = -\int p(x) \log p(x) \, dx$

For a Gaussian $\mathcal{N}(\mu, \sigma^2)$:

$h = \frac{1}{2}\log(2\pi e \sigma^2)$

Implementation

p = [0.5, 0.5] for fair coin

1import numpy as np
2
3# Discrete probability distribution
4p = np.array([0.5, 0.5])  # Fair coin
5
6# Compute entropy in bits (log base 2)
7def entropy(p):
8    mask = p > 0  # Handle zeros
9    return -np.sum(p[mask] * np.log2(p[mask]))
10
11H = entropy(p)
12print(f"Entropy of fair coin: {H:.4f} bits")  # Output: 1.0000
13
14# Compare with biased coin
15p_biased = np.array([0.9, 0.1])
16H_biased = entropy(p_biased)
17print(f"Entropy of biased coin: {H_biased:.4f} bits")  # Output: 0.4690

Cross-Entropy

Definition

The cross-entropy between two distributions P and Q is:

$H(P, Q) = -\sum_{x} p(x) \log q(x) = \mathbb{E}_{x \sim P}\left[-\log Q(x)\right]$

Cross-entropy measures the average number of bits needed to encode samples from P using a code optimized for Q.

The Key Relationship

Cross-entropy decomposes into entropy plus KL divergence:

$H(P, Q) = H(P) + D_{KL}(P \| Q)$

Since $D_{KL} \geq 0$, we have $H(P, Q) \geq H(P)$. Using the "wrong" distribution Q always costs extra bits.

Connection to Maximum Likelihood

For a dataset $\{x_1, \ldots, x_n\}$ from distribution P, the cross-entropy is:

$H(P, Q) \approx -\frac{1}{n}\sum_{i=1}^n \log q(x_i)$

This is the negative log-likelihood! Minimizing cross-entropy equals maximum likelihood estimation.

KL Divergence

Definition

The Kullback-Leibler divergence from Q to P (read "KL from P to Q" or "KL of P relative to Q") is:

$D_{KL}(P \| Q) = \sum_{x} p(x) \log \frac{p(x)}{q(x)} = \mathbb{E}_{x \sim P}\left[\log \frac{P(x)}{Q(x)}\right]$

For continuous distributions:

$D_{KL}(P \| Q) = \int p(x) \log \frac{p(x)}{q(x)} \, dx$

Properties of KL Divergence

1. Non-negativity: $D_{KL}(P \| Q) \geq 0$ with equality if and only if P = Q almost everywhere. (Gibbs' inequality)
2. Not symmetric: $D_{KL}(P \| Q) \neq D_{KL}(Q \| P)$ in general. KL is not a true distance metric.
3. Not a metric: Beyond asymmetry, KL doesn't satisfy the triangle inequality.
4. Requires absolute continuity: If $q(x) = 0$ where$p(x) > 0$, then $D_{KL} = \infty$.

Forward vs Reverse KL

The asymmetry of KL divergence has profound implications:

KL Divergence for Gaussians

For two univariate Gaussians $P = \mathcal{N}(\mu_1, \sigma_1^2)$ and$Q = \mathcal{N}(\mu_2, \sigma_2^2)$:

$D_{KL}(P \| Q) = \log\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} - \frac{1}{2}$

For multivariate Gaussians with covariance matrices $\Sigma_1, \Sigma_2$:

$D_{KL}(P \| Q) = \frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - d + \text{tr}(\Sigma_2^{-1}\Sigma_1) + (\mu_2 - \mu_1)^\top\Sigma_2^{-1}(\mu_2 - \mu_1)\right]$

Implementation

Useful when closed form is unavailable

1import torch
2from torch.distributions import Normal, kl_divergence
3
4# Define two Gaussian distributions
5P = Normal(loc=0.0, scale=1.0)  # True distribution
6Q = Normal(loc=1.0, scale=2.0)  # Approximation
7
8# Closed-form KL divergence
9kl_closed = kl_divergence(P, Q)
10print(f"KL(P||Q) closed-form: {kl_closed.item():.4f}")
11
12# Monte Carlo estimate of KL
13samples = P.sample((100000,))
14log_p = P.log_prob(samples)
15log_q = Q.log_prob(samples)
16kl_mc = (log_p - log_q).mean()
17print(f"KL(P||Q) Monte Carlo: {kl_mc.item():.4f}")
18
19# Note the asymmetry!
20kl_reverse = kl_divergence(Q, P)
21print(f"KL(Q||P): {kl_reverse.item():.4f}")

Mutual Information

Definition

Mutual information measures how much information one random variable contains about another:

$I(X; Y) = D_{KL}(P_{XY} \| P_X \otimes P_Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)p(y)}$

Equivalently:

$I(X; Y) = H(X) - H(X|Y) = H(Y) - H(Y|X) = H(X) + H(Y) - H(X, Y)$

Properties

• Non-negativity: $I(X; Y) \geq 0$
• Symmetry: $I(X; Y) = I(Y; X)$ (unlike KL!)
• Zero for independence: $I(X; Y) = 0$ iff X and Y are independent
• Self-information: $I(X; X) = H(X)$

Information Theory in Diffusion

Now let's see how these concepts appear in diffusion models:

The Variational Lower Bound (ELBO)

The training objective for diffusion models is derived from the ELBO:

$\log p(x_0) \geq \mathbb{E}_q\left[\log \frac{p(x_{0:T})}{q(x_{1:T}|x_0)}\right] = \text{ELBO}$

This can be decomposed into three terms:

Information Flow in Diffusion

Think of diffusion as information flow:

1. Forward process: Information about $x_0$ is progressively destroyed. By $t = T$, almost no information remains ($I(x_0; x_T) \approx 0$).
2. Reverse process: The neural network must recreate information. Each step recovers some mutual information with $x_0$.
3. Training: We teach the network to maximally recover information at each step by matching the true reverse posterior.

Entropy Throughout the Process

The entropy changes systematically through the diffusion process:

• At t=0: $H(x_0)$ is the entropy of the data distribution (relatively low for natural images)
• At t=T: $H(x_T) \approx H(\mathcal{N}(0, I))$ (maximum for fixed variance)
• During forward process: Entropy generally increases (adding noise adds uncertainty)
• During reverse process: Entropy decreases as structure is recovered

Summary

In this section, we covered the essential information theory concepts for diffusion models:

1. Entropy $H(X)$: Measures uncertainty/information content. Maximum for uniform distributions; Gaussians maximize entropy for fixed variance.
2. Cross-entropy $H(P, Q)$: Cost of using wrong code. Equals entropy plus KL divergence. Foundation of classification losses.
3. KL divergence $D_{KL}(P \| Q)$: Measures distribution difference. Asymmetric! Forward KL is mode-covering; reverse KL is mode-seeking.
4. Mutual information $I(X; Y)$: Shared information between variables. Data processing inequality limits information recovery.
5. Diffusion connection: The ELBO is expressed as expectations and KL divergences. Training minimizes KL between true and learned reverse processes.

Exercises

Conceptual Questions

1. Why is KL divergence always non-negative? Provide an intuitive explanation using the coding interpretation.
2. Explain why minimizing cross-entropy $H(P, Q)$ with respect to Q is equivalent to minimizing $D_{KL}(P \| Q)$ when P is fixed.
3. In variational inference, we minimize $D_{KL}(Q \| P)$ rather than$D_{KL}(P \| Q)$. What practical reason requires this choice?
4. The forward diffusion process increases entropy. Does this mean we're "adding information" to the image? Explain carefully.

Computational Exercises

1. Compute the entropy of a categorical distribution with K outcomes and probability$p_k = k/Z$ where $Z = \sum_{k=1}^K k$. How does entropy scale with K?
2. Implement KL divergence for multivariate Gaussians using both the closed-form formula and Monte Carlo sampling. Compare accuracy vs. number of samples.
3. Given $p(x) = \mathcal{N}(0, 1)$ and$q(x) = 0.5\mathcal{N}(-2, 1) + 0.5\mathcal{N}(2, 1)$ (a mixture), estimate $D_{KL}(P \| Q)$ using Monte Carlo. What happens and why?
4. For a simple 2-state diffusion process, compute the mutual information$I(x_0; x_t)$ as a function of t. Verify it decreases monotonically.

Challenge Problem

Derive the decomposition of the ELBO for diffusion models:

1. Start with $\log p(x_0) = \log \int p(x_{0:T}) dx_{1:T}$
2. Apply Jensen's inequality with variational distribution $q(x_{1:T}|x_0)$
3. Expand using the chain rule of probability
4. Identify the three terms: reconstruction, prior matching, and denoising matching
5. Show that the denoising term can be written as a sum of KL divergences