Marginal and Conditional Distributions

Learning Objectives

By the end of this section, you will master two fundamental operations on joint distributions that are essential for probabilistic reasoning and machine learning. You will be able to:

1. Define and compute marginal distributions from joint distributions, understanding how to "integrate out" or "sum out" a variable
2. Define and compute conditional distributions, understanding how observing one variable changes our beliefs about another
3. Explain the formulas for marginal and conditional distributions, interpreting every symbol and understanding what each operation measures
4. Visualize how slicing a joint distribution gives conditionals and how projecting gives marginals
5. Connect conditionals to Bayes' theorem, understanding the relationship between $P(Y|X)$ and $P(X|Y)$
6. Apply these concepts to real-world problems: medical diagnosis, prediction, and Bayesian inference
7. Recognize how marginal and conditional distributions are fundamental to modern AI: neural network outputs, generative models, and probabilistic programming
8. Implement marginal and conditional computations in Python with NumPy and SciPy

The Big Picture: Two Fundamental Operations

"To understand the world, we must learn to ignore some things and focus on others." — The essence of marginal and conditional distributions

Given a joint distribution $f_{X,Y}(x,y)$, we can perform two fundamental operations:

1. Marginalization: Ignoring a Variable

Marginal distribution answers: "What is the distribution of $X$ if we don't care about $Y$?"

Intuition: We "sum up" or "integrate out" the variable we don't care about. It's like looking at a shadow of the 2D joint distribution projected onto the X-axis.

2. Conditioning: Focusing Given Information

Conditional distribution answers: "Given that we observed $Y=y$, what is the distribution of $X$?"

Intuition: We "slice" the joint distribution at the observed value and renormalize to get a valid distribution. It's like looking at a cross-section of the 2D joint distribution.

Interactive 3D Exploration

Before diving into the formal mathematics, let's build intuition by exploring these concepts in 3D. The visualization below shows a bivariate normal distribution as a surface, with marginal distributions projected onto the walls and a conditional slice through the surface.

The Historical Story: From Gambling to Scientific Reasoning

The concepts of marginal and conditional distributions emerged gradually over centuries of wrestling with probabilistic reasoning.

The Origins: Thomas Bayes (1701-1761)

Reverend Thomas Bayes tackled the inverse probability problem: Given that we observe some data, what can we infer about the underlying cause? This is precisely the question conditional distributions answer.

Bayes' famous theorem (published posthumously in 1763) relates $P(A|B)$ to $P(B|A)$:

The denominator $P(B)$ is a marginal probability, computed by summing over all possible values of $A$.

Formalization: Laplace and Beyond

Pierre-Simon Laplace (1749-1827) extended Bayes' ideas, showing how to update beliefs about unknown parameters given data. His work required careful manipulation of joint, marginal, and conditional distributions.

Karl Pearson (1857-1936) and R.A. Fisher (1890-1962) developed the mathematical theory of multivariate distributions, formalizing marginal and conditional distributions in their modern form.

Marginal Distributions: Extracting Single-Variable Behavior

Mathematical Definition

The marginal distribution of $X$ is obtained by integrating (or summing) the joint distribution over all possible values of $Y$:

Let's unpack this formula:

• $f_X(x)$: The marginal PDF of $X$ alone, a function of $x$ only
• $f_{X,Y}(x, y)$: The joint PDF, a function of both $x$ and $y$
• $int_{-infty}^{infty} cdots dy$: Integration over all possible values of $Y$, "summing out" the $Y$ variable

For discrete random variables:

where $mathcal{Y}$ is the set of all possible values of $Y$.

Example: Bivariate Normal

For a bivariate normal (X, Y) \sim N(oldsymbol{\mu}, oldsymbol{\Sigma}) with:

The marginal distributions are:

Key insight: The marginal distributions are just univariate normals with the original means and variances! The correlation $ho$ does not appear in the marginals—it only affects the joint behavior.

Visualizing Marginals: Integration in Action

The best way to understand marginalization is to visualize it. The marginal at a specific $x$ value is the area under the joint PDF along the corresponding vertical line.

In the visualization above:

• The left heatmap shows the joint PDF $f(x,y)$
• The orange column highlights all points with that $x$ value
• The marginal value $f_X(x)$ is the sum (integral) of all the density in that column
• Doing this for all $x$ values gives the marginal curve

Conditional Distributions: Updating Beliefs Given Information

Mathematical Definition

The conditional distribution of $X$ given $Y = y$ is:

Let's understand each part:

• $f_{X|Y}(x|y)$: The conditional PDF of $X$ given that we observed $Y = y$
• $f_{X,Y}(x, y)$: The joint PDF evaluated at $(x, y)$
• $f_Y(y)$: The marginal PDF of $Y$, which serves as the normalizing constant

For discrete random variables:

Example: Conditional of Bivariate Normal

For the bivariate normal, the conditional $X | Y = y$ is also normal:

This elegant formula tells us exactly how observing $Y$ affects our beliefs about $X$:

1. Conditional Mean: E[X|Y=y] = mu_X + ho rac{sigma_X}{sigma_Y}(y - mu_Y)

   The mean shifts proportionally to how far $y$ is from its mean $mu_Y$. If $y > mu_Y$ and $ho > 0$, we expect $X$ to be above its mean too!

2. Conditional Variance: $ext{Var}(X|Y=y) = sigma_X^2(1 - ho^2)$

   The variance is reduced by a factor of $(1 - ho^2)$. Higher correlation means more variance reduction—knowing $Y$ gives us more information about $X$!

3. Still Normal: The conditional distribution remains Gaussian

   This is a special property of the multivariate normal—conditionals are always normal. Not all distributions have this nice property!

Visualizing Conditionals: Slicing the Joint Distribution

The conditional distribution is like taking a "slice" through the joint distribution at a fixed value of one variable. Let's see this in action:

In the visualization above:

• The purple curve is the raw slice through the joint PDF at the selected $X$ value
• This slice is not normalized—it doesn't integrate to 1
• The orange curve is the true conditional $f(y|x)$, which is the normalized version
• The red dashed line shows the conditional mean $E[Y|X=x] = ho x$
• The green dashed lines show ±1 conditional standard deviation

Discrete Example: Interactive Joint, Marginal, and Conditional

For discrete random variables, we can visualize the relationships with probability tables. Hover over cells to see how marginal and conditional distributions are computed:

Key observations from the discrete example:

• Marginal of X (green column): Sum each row across all Y values
• Marginal of Y (green row): Sum each column across all X values
• Conditional P(Y|X=x): Divide each cell in row x by the marginal P(X=x)
• Conditional P(X|Y=y): Divide each cell in column y by the marginal P(Y=y)
• Conditionals always sum to 1 across the variable being predicted

Connection to Bayes' Theorem: The Two Directions of Conditioning

A crucial insight is that we can condition in two directions:

Both formulas use the same joint $f_{X,Y}(x,y)$ in the numerator, but different marginals in the denominator. This gives us Bayes' Theorem:

In words:

Posterior = (Likelihood × Prior) / Evidence

This formula allows us to "invert" conditional probabilities—if we know $P(Y|X)$ (often easier to estimate), we can compute $P(X|Y)$ (often what we want to know).

Key Formulas Summary

Here is a comprehensive reference for the formulas introduced in this section:

Bivariate Normal Special Case

AI/ML Applications: Why Every Engineer Needs These Concepts

1. Classification as Conditional Probability

Every classifier computes a conditional probability:

The softmax output of a neural network is exactly this conditional distribution over classes given the input features. The class with highest $P(Y=k|X)$ is the prediction.

2. Generative Models and the Product Rule

Generative models like GPT factor the joint distribution of tokens using the product rule:

Each term is a conditional distribution of the next token given previous tokens. The language model learns these conditionals!

3. Latent Variable Models

In VAEs and many other models, we have observable data $X$ and hidden/latent variables $Z$:

This is marginalization over the latent space. The ELBO (Evidence Lower Bound) in VAEs is derived from this marginal likelihood.

4. Bayesian Deep Learning

In Bayesian neural networks, weights $W$ are random variables. After observing data $D$:

This is conditioning on the data. The denominator $p(D)$ is a marginal likelihood:

5. Regression as Conditional Expectation

The goal of regression is to estimate $E[Y|X=x]$, the conditional mean of $Y$ given $X$. For bivariate normal:

This is exactly linear regression! The regression coefficient eta_1 = ho rac{sigma_Y}{sigma_X} and $R^2 = ho^2$ is the fraction of variance explained.

Python Implementation

Computing Marginal Distributions

1import numpy as np
2from scipy.stats import multivariate_normal
3import matplotlib.pyplot as plt
4
5# Create a bivariate normal distribution
6# Mean vector and covariance matrix
7mean = [0, 0]
8cov = [[1, 0.7],    # Var(X)=1, Cov(X,Y)=0.7
9       [0.7, 1]]     # Cov(X,Y)=0.7, Var(Y)=1
10
11# Create the joint distribution
12joint_dist = multivariate_normal(mean=mean, cov=cov)
13
14# Generate grid for visualization
15x = np.linspace(-3, 3, 100)
16y = np.linspace(-3, 3, 100)
17X, Y = np.meshgrid(x, y)
18pos = np.dstack((X, Y))
19
20# Evaluate joint PDF
21joint_pdf = joint_dist.pdf(pos)
22
23# Compute marginal of X by integrating out Y
24# f_X(x) = integral of f(x,y) dy
25dx = x[1] - x[0]
26marginal_x = np.trapz(joint_pdf, y, axis=0)
27
28# For bivariate normal, marginal is N(mu_x, sigma_x^2)
29# True marginal should be N(0, 1) - standard normal
30from scipy.stats import norm
31true_marginal = norm.pdf(x, loc=0, scale=1)
32
33# Verify: our numerical integration matches true marginal
34error = np.max(np.abs(marginal_x - true_marginal))
35print(f"Max error between numerical and true marginal: {error:.6f}")
36
37# Plot comparison
38plt.figure(figsize=(10, 4))
39plt.subplot(1, 2, 1)
40plt.plot(x, marginal_x, 'b-', linewidth=2, label='Numerical')
41plt.plot(x, true_marginal, 'r--', linewidth=2, label='Analytical')
42plt.xlabel('x')
43plt.ylabel('Density')
44plt.title('Marginal Distribution of X')
45plt.legend()
46
47plt.subplot(1, 2, 2)
48plt.contour(X, Y, joint_pdf, levels=10)
49plt.xlabel('X')
50plt.ylabel('Y')
51plt.title('Joint Distribution')
52plt.tight_layout()

Computing Conditional Distributions

1import numpy as np
2from scipy.stats import norm, multivariate_normal
3
4# Bivariate normal parameters
5mu_x, mu_y = 0, 0
6sigma_x, sigma_y = 1, 1
7rho = 0.7  # Correlation coefficient
8
9# Given X = x_obs, compute conditional distribution Y | X = x_obs
10def conditional_y_given_x(y, x_obs, mu_x, mu_y, sigma_x, sigma_y, rho):
11    """
12    Compute f(y | X = x_obs) for bivariate normal.
13
14    The conditional is also normal with:
15    - Mean: mu_Y|X = mu_y + rho * (sigma_y/sigma_x) * (x_obs - mu_x)
16    - Variance: sigma^2_Y|X = sigma_y^2 * (1 - rho^2)
17    """
18    # Conditional mean
19    cond_mean = mu_y + rho * (sigma_y / sigma_x) * (x_obs - mu_x)
20
21    # Conditional standard deviation
22    cond_std = sigma_y * np.sqrt(1 - rho**2)
23
24    # Return conditional PDF
25    return norm.pdf(y, loc=cond_mean, scale=cond_std)
26
27# Example: What is f(Y | X = 1.5)?
28x_obs = 1.5
29y_values = np.linspace(-3, 3, 100)
30
31# Compute conditional distribution
32cond_pdf = conditional_y_given_x(y_values, x_obs, mu_x, mu_y,
33                                  sigma_x, sigma_y, rho)
34
35# Key statistics
36cond_mean = mu_y + rho * (sigma_y / sigma_x) * (x_obs - mu_x)
37cond_std = sigma_y * np.sqrt(1 - rho**2)
38
39print(f"Given X = {x_obs}:")
40print(f"  Conditional mean E[Y|X={x_obs}] = {cond_mean:.3f}")
41print(f"  Conditional std dev = {cond_std:.3f}")
42print(f"  Original Y mean = {mu_y}, std = {sigma_y}")
43print(f"  Variance reduction = {(1 - cond_std**2/sigma_y**2)*100:.1f}%")
44
45# Verify using Bayes' rule: f(y|x) = f(x,y) / f(x)
46joint = multivariate_normal(mean=[mu_x, mu_y],
47                            cov=[[sigma_x**2, rho*sigma_x*sigma_y],
48                                 [rho*sigma_x*sigma_y, sigma_y**2]])
49
50# Numerically verify for a specific point
51y_test = 1.0
52f_xy = joint.pdf([x_obs, y_test])
53f_x = norm.pdf(x_obs, loc=mu_x, scale=sigma_x)
54f_y_given_x = f_xy / f_x
55
56print(f"\nVerification at y={y_test}:")
57print(f"  f(y|x) from formula: {cond_pdf[50]:.6f}")
58print(f"  f(x,y)/f(x): {f_y_given_x:.6f}")

Bayes' Rule in Practice

1import numpy as np
2
3# Medical test scenario: demonstrating conditional distributions
4# X = Disease status (0=healthy, 1=sick)
5# Y = Test result (0=negative, 1=positive)
6
7# Parameters
8prevalence = 0.05      # P(sick) = 5%
9sensitivity = 0.90     # P(positive | sick) = 90%
10specificity = 0.95     # P(negative | healthy) = 95%
11
12# Build joint distribution table
13# P(X=0) = 0.95, P(X=1) = 0.05
14# P(Y=0|X=0) = 0.95, P(Y=1|X=0) = 0.05
15# P(Y=0|X=1) = 0.10, P(Y=1|X=1) = 0.90
16
17joint = np.array([
18    [0.95 * specificity, 0.95 * (1-specificity)],   # Healthy
19    [0.05 * (1-sensitivity), 0.05 * sensitivity]    # Sick
20])
21
22print("Joint Distribution P(X,Y):")
23print(f"{'':>15} {'Y=0 (Test-)':>12} {'Y=1 (Test+)':>12} {'P(X)':>10}")
24print(f"{'X=0 (Healthy)':>15} {joint[0,0]:>12.4f} {joint[0,1]:>12.4f} {joint[0].sum():>10.4f}")
25print(f"{'X=1 (Sick)':>15} {joint[1,0]:>12.4f} {joint[1,1]:>12.4f} {joint[1].sum():>10.4f}")
26print(f"{'P(Y)':>15} {joint[:,0].sum():>12.4f} {joint[:,1].sum():>12.4f}")
27
28# Marginal distributions
29p_x = joint.sum(axis=1)  # P(X=0), P(X=1)
30p_y = joint.sum(axis=0)  # P(Y=0), P(Y=1)
31
32# Conditional: P(Y|X) - forward direction
33print("\n--- Forward Conditional P(Y|X) ---")
34p_y_given_x = joint / p_x[:, np.newaxis]
35print(f"P(Test+ | Healthy) = {p_y_given_x[0,1]:.4f}  (False positive rate)")
36print(f"P(Test+ | Sick) = {p_y_given_x[1,1]:.4f}     (Sensitivity)")
37
38# Conditional: P(X|Y) - inverse direction (using Bayes!)
39print("\n--- Inverse Conditional P(X|Y) via Bayes ---")
40p_x_given_y = joint / p_y[np.newaxis, :]
41print(f"P(Sick | Test+) = {p_x_given_y[1,1]:.4f}     (Positive predictive value)")
42print(f"P(Sick | Test-) = {p_x_given_y[1,0]:.4f}    (1 - NPV)")
43
44# Bayes' Rule verification
45print("\n--- Bayes' Rule Verification ---")
46# P(Sick | Test+) = P(Test+ | Sick) * P(Sick) / P(Test+)
47bayes_result = (sensitivity * prevalence) / p_y[1]
48print(f"P(Sick|Test+) via Bayes = {bayes_result:.4f}")
49print(f"P(Sick|Test+) from table = {p_x_given_y[1,1]:.4f}")
50
51# Key insight
52print("\n--- Key Insight: Base Rate Fallacy ---")
53print(f"Even with 90% sensitivity and 95% specificity,")
54print(f"P(Actually sick | Positive test) = only {p_x_given_y[1,1]*100:.1f}%!")
55print(f"This is because disease prevalence is only {prevalence*100}%.")

Common Pitfalls and Misconceptions

Summary: What You've Mastered

Congratulations! You now understand two of the most fundamental operations in probability and statistics:

Marginal Distributions

• Definition: $f_X(x) = int f_{X,Y}(x,y) , dy$
• Intuition: "Ignore Y, what's the distribution of X alone?"
• Computation: Integrate/sum out the variable you don't care about
• Key insight: Marginals don't reveal correlation or dependence

Conditional Distributions

• Definition: f_{X|Y}(x|y) = rac{f_{X,Y}(x,y)}{f_Y(y)}
• Intuition: "Given that I observed Y=y, what's the distribution of X?"
• Computation: Slice the joint at the observed value, then normalize
• Key insight: Conditional mean shifts; conditional variance often shrinks

Bayes' Theorem

• Formula: P(X|Y) = rac{P(Y|X) P(X)}{P(Y)}
• Purpose: Convert between $P(Y|X)$ and $P(X|Y)$
• Warning: Base rates matter! Don't confuse these two directions

Next Steps

In the next sections of this chapter, we will build on these foundations to study:

1. Covariance and Correlation: Quantifying the strength and direction of linear dependence
2. Covariance Matrix: Extending to high-dimensional vectors
3. Multivariate Normal: The workhorse distribution for multivariate analysis
4. Conditional Distributions of MVN: Gaussian conditioning and its applications

Interactive Summary

Explore the complete relationship between joint, marginal, and conditional distributions for the bivariate normal: