Conditional Distributions of MVN

Learning Objectives

Conditional distributions of the multivariate normal (MVN) are among the most powerful and widely-used results in all of statistics and machine learning. By the end of this section, you will:

1. Master the formulas for conditional mean and conditional covariance of the multivariate normal distribution, understanding every term intuitively
2. Visualize how conditioning on observed variables "slices" the joint distribution, producing a new Gaussian with shifted mean and reduced variance
3. Derive the deep connection between conditional MVN and linear regression, showing that regression coefficients emerge naturally from the covariance structure
4. Understand how Gaussian Processes use conditional MVN for function-space inference, enabling uncertainty-aware predictions
5. Recognize the Kalman filter as sequential conditioning on observations, making state estimation principled
6. Apply these concepts to modern deep learning: variational inference, latent variable models, and Bayesian neural networks
7. Implement conditional MVN computations efficiently in Python using matrix operations

The Big Picture: What Happens When We Condition

"Conditioning is the soul of statistics." — Joe Blitzstein

Consider a multivariate normal distribution over several variables. When we observe (condition on) some of these variables, what happens to the distribution of the remaining variables?

The remarkable answer for the multivariate normal is:

This is extraordinary! For most distributions, conditioning leads to complex, non-standard forms. But for the MVN, we get clean, closed-form formulas that are computationally tractable.

The Geometric Intuition

Imagine a 2D Gaussian as a "probability cloud" shaped like an ellipse. When we condition on $X = x_0$:

1. We "slice" the cloud with a vertical plane at $x = x_0$
2. The cross-section is another Gaussian (1D in this case)
3. If the original ellipse was tilted (correlated), the slice is centered away from the marginal mean of Y
4. The slice is always narrower than the full marginal distribution of Y

Historical Context: From Gauss to Machine Learning

The properties of conditional Gaussian distributions have a rich history spanning over 200 years.

Carl Friedrich Gauss (1777-1855)

Gauss developed the method of least squares for astronomical observations, implicitly using properties of conditional normal distributions. His work on error analysis laid the foundation for understanding how correlated measurements could be optimally combined.

Francis Galton (1822-1911)

Galton discovered regression toward the mean while studying heredity. He noticed that tall parents tended to have children closer to average height. This is precisely the behavior of the conditional mean in a bivariate normal:

When $|\rho| < 1$, the conditional mean is "regressed" toward $\mu_Y$ compared to naive prediction.

Karl Pearson (1857-1936)

Pearson formalized the correlation coefficient and developed the full theory of the bivariate normal distribution, including the explicit formulas for conditional distributions that we use today.

Rudolf Kálmán (1930-2016)

Kálmán revolutionized control theory by showing that optimal state estimation in linear dynamical systems is achieved by sequential Gaussian conditioning. The Kalman filter, published in 1960, uses the MVN conditional formulas at every time step.

Bivariate Normal Conditioning: The Foundation

Let's start with the simplest case: two correlated normal random variables $(X, Y)$ with joint distribution:

where $\rho \in [-1, 1]$ is the correlation coefficient.

The Conditional Distribution

Given that we observe $X = x$, the conditional distribution of $Y$ is:

where the conditional parameters are:

Conditional Mean

Let's understand each term:

• $\mu_Y$: The starting point—the unconditional mean of $Y$
• $(x - \mu_X)$: How far the observed $X$ is from its mean (in raw units)
• $(x - \mu_X)/\sigma_X$: How far $X$ is from its mean (in standard deviations)
• $\rho$: The correlation—how much $Y$ should "follow" $X$
• $\rho \sigma_Y / \sigma_X$: The regression coefficient—how many units $Y$ shifts per unit increase in $X$

Conditional Variance

This formula encapsulates a profound insight:

• The conditional variance does not depend on x—the uncertainty reduction is the same regardless of which $X$ value we observe
• The factor $(1 - \rho^2)$ is always between 0 and 1, so variance is always reduced
• When $\rho = 0$: $\sigma_{Y|X}^2 = \sigma_Y^2$ (no reduction—$X$ provides no information)
• When $|\rho| = 1$: $\sigma_{Y|X}^2 = 0$ (perfect prediction—$Y$ is deterministic given $X$)
• $\rho^2$ is the fraction of variance explained—this is exactly $R^2$ in linear regression!

Interactive Exploration: See Conditioning in Action

The visualization below shows a bivariate normal distribution. Adjust the correlation and the observed X value to see how the conditional distribution of Y changes.

General Multivariate Normal Conditioning

Now let's extend to the general case with $n$ variables. We partition the random vector into two groups:

where:

• $\mathbf{X}_1$: The variables we want to predict (dimension $p$)
• $\mathbf{X}_2$: The observed (conditioning) variables (dimension $q$)
• $\boldsymbol{\Sigma}_{11}$: Covariance of $\mathbf{X}_1$ ($p \times p$)
• $\boldsymbol{\Sigma}_{22}$: Covariance of $\mathbf{X}_2$ ($q \times q$)
• $\boldsymbol{\Sigma}_{12} = \boldsymbol{\Sigma}_{21}^T$: Cross-covariance ($p \times q$)

The General Conditional Formulas

Given observation $\mathbf{X}_2 = \mathbf{x}_2$, the conditional distribution of $\mathbf{X}_1$ is:

where:

Understanding the Matrix Formulas

Let's interpret each term:

Conditional Mean

• $\boldsymbol{\mu}_1$: Start from the unconditional mean of $\mathbf{X}_1$
• $\boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1}$: The multivariate regression coefficient matrix ($p \times q$)
• $(\mathbf{x}_2 - \boldsymbol{\mu}_2)$: How far the observations deviate from their mean
• The mean shifts linearly with the observed values

Conditional Covariance

• $\boldsymbol{\Sigma}_{11}$: Start from the unconditional covariance of $\mathbf{X}_1$
• $\boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} \boldsymbol{\Sigma}_{21}$: Variance reduction from knowing $\mathbf{X}_2$
• This is a positive semi-definite matrix, so $\bar{\boldsymbol{\Sigma}} \preceq \boldsymbol{\Sigma}_{11}$
• The result is independent of the specific observed values $\mathbf{x}_2$

Connection to Linear Regression

One of the most beautiful results in statistics is that linear regression emerges naturally from the conditional MVN.

The Deep Connection

Consider the bivariate normal $(X, Y)$. The conditional expectation is:

This is exactly the linear regression line $E[Y|X] = \beta_0 + \beta_1 X$!

Interactive Linear Regression Demo

The visualization below shows data from a bivariate normal and the regression line. The conditional distribution at any X value is shown as an orange curve.

Gaussian Processes: MVN Conditioning in Function Space

"A Gaussian Process is a distribution over functions, where any finite collection of function values is jointly Gaussian."

Gaussian Processes (GPs) are one of the most powerful applications of conditional MVN. The key insight: GP regression is just MVN conditioning in infinite dimensions.

How GPs Work

A GP defines a prior distribution over functions $f(x)$. When we observe training data $(X, \mathbf{y})$, we condition on these observations to get a posterior distribution over functions.

At any finite set of test points $X_*$, the joint distribution of function values is:

where $K(\cdot, \cdot)$ is the kernel function that defines the prior covariance structure.

The GP Posterior (Conditional MVN!)

Given observations $\mathbf{y} = \mathbf{f} + \boldsymbol{\epsilon}$ (with noise $\boldsymbol{\epsilon} \sim N(\mathbf{0}, \sigma_n^2 I)$), the posterior is:

where:

These are exactly the conditional MVN formulas! The kernel defines $\boldsymbol{\Sigma}_{12}, \boldsymbol{\Sigma}_{22}$, etc.

Kalman Filter: Sequential Conditioning

The Kalman filter is perhaps the most celebrated application of conditional Gaussians in engineering. It solves the problem of optimal state estimation in linear dynamical systems with Gaussian noise.

The Problem Setting

Consider a system with hidden state $\mathbf{x}_t$ that evolves according to:

where $\mathbf{w}_t \sim N(\mathbf{0}, Q)$ is process noise and $\mathbf{v}_t \sim N(\mathbf{0}, R)$ is measurement noise.

The Kalman Filter Algorithm

At each time step, the Kalman filter performs two operations:

1. Predict (Prior): Propagate the state estimate forward in time
   $$\hat{\mathbf{x}}_{t|t-1} = F \hat{\mathbf{x}}_{t-1|t-1}$$
   $$P_{t|t-1} = F P_{t-1|t-1} F^T + Q$$
2. Update (Condition!): Incorporate new measurement via MVN conditioning
   $$K_t = P_{t|t-1} H^T (H P_{t|t-1} H^T + R)^{-1} \quad \text{(Kalman gain)}$$
   $$\hat{\mathbf{x}}_{t|t} = \hat{\mathbf{x}}_{t|t-1} + K_t (\mathbf{z}_t - H \hat{\mathbf{x}}_{t|t-1})$$
   $$P_{t|t} = (I - K_t H) P_{t|t-1}$$

Deep Learning Applications

Conditional MVN is not just classical statistics—it's fundamental to modern deep learning.

1. Variational Autoencoders (VAEs)

VAEs use conditional Gaussians in two places:

• Encoder: $q(\mathbf{z}|\mathbf{x}) = N(\boldsymbol{\mu}_\phi(\mathbf{x}), \boldsymbol{\sigma}_\phi^2(\mathbf{x}))$ — maps data to latent distribution
• Decoder: $p(\mathbf{x}|\mathbf{z}) = N(\boldsymbol{\mu}_\theta(\mathbf{z}), \boldsymbol{\sigma}_\theta^2)$ — generates data from latent

The reparameterization trick leverages the fact that conditional Gaussians can be sampled as $\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon}$ where $\boldsymbol{\epsilon} \sim N(\mathbf{0}, I)$.

2. Normalizing Flows

Flow-based models transform a simple base distribution (often Gaussian) through invertible transformations. Affine coupling layers use conditional Gaussians:

3. Bayesian Neural Networks

In Bayesian deep learning, we place priors on weights and compute posteriors. The approximate posterior is often Gaussian:

Predictions involve conditioning on inputs, integrating out weights.

4. Attention Mechanisms

While standard attention isn't explicitly Gaussian, probabilistic attention mechanisms can be interpreted as computing conditional expectations over keys given queries, with Gaussian assumptions on representations.

5. Latent Variable Models

Many generative models (GMMs, HMMs, LDA) use conditional Gaussians for observations given latent states. The E-step in EM computes $p(\mathbf{z}|\mathbf{x})$, which for Gaussian mixtures is a weighted average of conditionals.

Python Implementation

Bivariate Normal Conditioning

1import numpy as np
2from scipy.stats import norm, multivariate_normal
3import matplotlib.pyplot as plt
4
5# Bivariate Normal Conditional Distribution
6# Given (X, Y) ~ N(mu, Sigma), find distribution of Y | X = x_obs
7
8# Parameters
9mu_x, mu_y = 0, 0
10sigma_x, sigma_y = 1, 1
11rho = 0.7  # Correlation coefficient
12
13# Observed value
14x_obs = 1.5
15
16# Conditional distribution formulas
17# Y | X = x_obs ~ N(mu_cond, sigma_cond^2)
18mu_cond = mu_y + rho * (sigma_y / sigma_x) * (x_obs - mu_x)
19sigma_cond = sigma_y * np.sqrt(1 - rho**2)
20
21print("Conditional Distribution Y | X = {:.2f}:".format(x_obs))
22print(f"  Mean: E[Y|X={x_obs}] = {mu_cond:.4f}")
23print(f"  Std Dev: sqrt(Var(Y|X)) = {sigma_cond:.4f}")
24print(f"  Variance Reduction: {rho**2 * 100:.1f}%")
25
26# Verify by computing from joint distribution
27cov_matrix = [[sigma_x**2, rho * sigma_x * sigma_y],
28              [rho * sigma_x * sigma_y, sigma_y**2]]
29
30# Condition using the formula: Sigma_12 @ Sigma_22^{-1}
31# For bivariate: beta = Sigma_12 / Sigma_22 = rho * sigma_y / sigma_x
32beta = cov_matrix[0][1] / cov_matrix[1][1]  # Note: roles swapped for Y|X
33print(f"\nVerification: regression coefficient = {beta:.4f}")
34
35# Compare with original Y distribution
36print(f"\nOriginal Y ~ N({mu_y}, {sigma_y**2})")
37print(f"Conditional Y|X ~ N({mu_cond:.3f}, {sigma_cond**2:.3f})")
38print(f"Knowing X reduces Y's variance by factor {1 - rho**2:.3f}")

General MVN Conditioning

1import numpy as np
2from scipy.stats import multivariate_normal
3
4# General MVN Conditional Distribution
5# Partition X = [X_1, X_2]^T with:
6#   X ~ N(mu, Sigma) where mu = [mu_1, mu_2], Sigma = [[S_11, S_12], [S_21, S_22]]
7# Then: X_1 | X_2 = x_2 ~ N(mu_bar, Sigma_bar)
8
9def mvn_conditional(mu, Sigma, x_2, indices_1, indices_2):
10    """
11    Compute conditional MVN distribution.
12
13    Parameters:
14    - mu: Full mean vector
15    - Sigma: Full covariance matrix
16    - x_2: Observed values for X_2
17    - indices_1: Indices of variables we want to condition TO
18    - indices_2: Indices of observed (conditioning) variables
19
20    Returns:
21    - mu_bar: Conditional mean
22    - Sigma_bar: Conditional covariance
23    """
24    mu = np.array(mu)
25    Sigma = np.array(Sigma)
26    x_2 = np.array(x_2)
27
28    # Extract submatrices
29    mu_1 = mu[indices_1]
30    mu_2 = mu[indices_2]
31
32    Sigma_11 = Sigma[np.ix_(indices_1, indices_1)]
33    Sigma_12 = Sigma[np.ix_(indices_1, indices_2)]
34    Sigma_21 = Sigma[np.ix_(indices_2, indices_1)]
35    Sigma_22 = Sigma[np.ix_(indices_2, indices_2)]
36
37    # Compute conditional parameters
38    Sigma_22_inv = np.linalg.inv(Sigma_22)
39
40    # Conditional mean: mu_1 + Sigma_12 @ Sigma_22^{-1} @ (x_2 - mu_2)
41    mu_bar = mu_1 + Sigma_12 @ Sigma_22_inv @ (x_2 - mu_2)
42
43    # Conditional covariance: Sigma_11 - Sigma_12 @ Sigma_22^{-1} @ Sigma_21
44    Sigma_bar = Sigma_11 - Sigma_12 @ Sigma_22_inv @ Sigma_21
45
46    return mu_bar, Sigma_bar
47
48# Example: 4D MVN, condition on variables 2 and 3
49mu = [0, 1, 2, 3]
50Sigma = [
51    [1.0, 0.5, 0.3, 0.1],
52    [0.5, 2.0, 0.7, 0.2],
53    [0.3, 0.7, 1.5, 0.4],
54    [0.1, 0.2, 0.4, 1.0]
55]
56
57# Observe X_2 = 0.5, X_3 = 1.0 (indices 1 and 2)
58x_observed = [0.5, 1.0]
59indices_1 = [0, 3]  # Variables to predict (X_1, X_4)
60indices_2 = [1, 2]  # Observed variables (X_2, X_3)
61
62mu_cond, Sigma_cond = mvn_conditional(mu, Sigma, x_observed, indices_1, indices_2)
63
64print("Conditional Distribution [X_1, X_4] | [X_2=0.5, X_3=1.0]:")
65print(f"  Conditional mean: {mu_cond}")
66print(f"  Conditional covariance:\\n{Sigma_cond}")
67
68# Verify: Create full distribution and sample
69full_dist = multivariate_normal(mean=mu, cov=Sigma)
70samples = full_dist.rvs(size=10000)
71
72# Filter samples close to observed values
73mask = (np.abs(samples[:, 1] - 0.5) < 0.1) & (np.abs(samples[:, 2] - 1.0) < 0.1)
74filtered = samples[mask][:, [0, 3]]
75
76print(f"\nEmpirical (from sampling):")
77print(f"  Mean: {np.mean(filtered, axis=0)}")
78print(f"  Covariance:\\n{np.cov(filtered.T)}")

Gaussian Process Implementation

1import numpy as np
2from scipy.stats import multivariate_normal
3import matplotlib.pyplot as plt
4
5# Gaussian Process = Infinite-dimensional MVN
6# The key insight: GP posterior is just conditional MVN
7
8def rbf_kernel(x1, x2, length_scale=1.0, variance=1.0):
9    """RBF (Squared Exponential) kernel."""
10    return variance * np.exp(-0.5 * (x1 - x2)**2 / length_scale**2)
11
12def gp_posterior(X_train, y_train, X_test,
13                 length_scale=1.0, variance=1.0, noise=0.1):
14    """
15    Compute GP posterior using conditional MVN formulas.
16
17    Parameters:
18    - X_train: Training inputs (n x 1)
19    - y_train: Training outputs (n,)
20    - X_test: Test inputs (m x 1)
21    - length_scale, variance: Kernel hyperparameters
22    - noise: Observation noise variance
23
24    Returns:
25    - mu_star: Posterior mean at test points
26    - cov_star: Posterior covariance at test points
27    """
28    n = len(X_train)
29    m = len(X_test)
30
31    # Build covariance matrices
32    K = np.zeros((n, n))
33    K_star = np.zeros((m, n))
34    K_star_star = np.zeros((m, m))
35
36    for i in range(n):
37        for j in range(n):
38            K[i, j] = rbf_kernel(X_train[i], X_train[j], length_scale, variance)
39            if i == j:
40                K[i, j] += noise  # Add noise on diagonal
41
42    for i in range(m):
43        for j in range(n):
44            K_star[i, j] = rbf_kernel(X_test[i], X_train[j], length_scale, variance)
45
46    for i in range(m):
47        for j in range(m):
48            K_star_star[i, j] = rbf_kernel(X_test[i], X_test[j], length_scale, variance)
49
50    # Conditional MVN formulas!
51    # This is EXACTLY: mu_bar = mu + Sigma_12 @ Sigma_22^{-1} @ (x - mu)
52    # With mu = 0 (GP prior mean) and x = y_train
53    K_inv = np.linalg.inv(K)
54
55    # Posterior mean: K_* @ K^{-1} @ y
56    mu_star = K_star @ K_inv @ y_train
57
58    # Posterior covariance: K_** - K_* @ K^{-1} @ K_*^T
59    cov_star = K_star_star - K_star @ K_inv @ K_star.T
60
61    return mu_star, cov_star
62
63# Example: Fit GP to noisy observations
64np.random.seed(42)
65X_train = np.array([-2, -1, 0, 1, 2.5])
66y_train = np.sin(X_train) + 0.1 * np.random.randn(len(X_train))
67
68X_test = np.linspace(-3, 3, 100)
69mu, cov = gp_posterior(X_train, y_train, X_test)
70std = np.sqrt(np.diag(cov))
71
72print("GP Posterior (at selected test points):")
73for i in [0, 25, 50, 75, 99]:
74    print(f"  x={X_test[i]:.2f}: mean={mu[i]:.3f}, std={std[i]:.3f}")
75
76print("\nKey insight: The GP posterior formulas are EXACTLY")
77print("the conditional MVN formulas!")

Common Pitfalls and Misconceptions

Summary: The Power of Gaussian Conditioning

You've now mastered one of the most powerful tools in probability and statistics. Let's recap:

The Key Formulas

Key Properties

1. Closure: Conditioning a Gaussian on another Gaussian yields a Gaussian
2. Linear mean: The conditional mean is a linear function of observed values
3. Constant variance: Conditional covariance doesn't depend on observed values
4. Variance reduction: Conditioning always reduces variance (information gain)

Applications Mastered

• Linear Regression: The regression line is $E[Y|X]$, and $R^2 = \rho^2$
• Gaussian Processes: GP posterior = conditional MVN in function space
• Kalman Filter: State estimation = sequential Gaussian conditioning
• Deep Learning: VAEs, flows, Bayesian NNs all use conditional Gaussians