Joint PMF and PDF

Learning Objectives

By the end of this section, you will have a deep, intuitive understanding of how multiple random variables interact. You will be able to:

1. Define joint probability mass functions (PMF) and probability density functions (PDF) for two or more random variables
2. Explain the fundamental difference between modeling variables separately vs. jointly, and why joint modeling captures critical information
3. Interpret every symbol in the joint PMF/PDF formulas and understand what each equation is measuring
4. Visualize joint distributions in 2D and 3D, understanding probability tables, heatmaps, and density surfaces
5. Distinguish between independent and dependent random variables using the factorization criterion
6. Calculate probabilities of events involving multiple variables by integrating or summing the joint distribution
7. Apply joint distributions to real-world scenarios: sports analytics, medical diagnosis, financial portfolios, and sensor fusion
8. Recognize how joint distributions are fundamental to modern AI: neural network activations, Bayesian networks, GANs, and multivariate regression
9. Implement joint PMF and PDF calculations in Python with NumPy and SciPy

Why Joint Distributions? The Fundamental Limitation of Univariate Thinking

"Everything is connected to everything else." — Leonardo da Vinci

When we study a single random variable $X$, we can describe its behavior completely with a PMF $p_X(x)$ or PDF $f_X(x)$. But nature rarely works in isolation. Consider these scenarios:

• Medical Diagnosis: Blood pressure and cholesterol aren't independent—high blood pressure often correlates with high cholesterol. Treating them as separate random variables misses critical information.
• Image Recognition: In a 28×28 pixel image (like MNIST), each pixel is a random variable. But pixels aren't independent—neighboring pixels have related intensities that form patterns (edges, shapes).
• Financial Portfolio: Stock A and Stock B may each have return distributions, but what matters for risk is their joint behavior. Do they crash together (positive correlation) or hedge each other (negative correlation)?
• Weather Forecasting: Temperature and humidity at a location are linked—knowing temperature gives information about likely humidity levels.

Interactive 3D Explorer: See Joint Distributions Come Alive

Before diving into the mathematical formalism, let's build intuition by playing with a real joint distribution. The visualization below shows a bivariate normal distribution—the most important joint distribution in statistics.

What you're seeing: The 3D surface represents the joint probability density $f(x, y)$. The height at each point $(x, y)$ shows how "likely" that combination is. The green curve on the back wall is the marginal distribution of X—what you get if you "ignore" Y. The blue curve on the left wall is the marginal distribution of Y.

The Historical Story: From Stars to Statistics

The concept of joint distributions emerged from astronomy in the late 18th and early 19th centuries. Pierre-Simon Laplace and Carl Friedrich Gauss were analyzing measurement errors in astronomical observations.

The Problem That Started It All

When measuring a star's position, astronomers made multiple observations that differed due to measurement errors. But here's the twist: errors in the horizontal and vertical directions weren't independent! Atmospheric refraction, telescope vibrations, and observer fatigue affected both dimensions simultaneously.

Gauss realized that to properly model measurement uncertainty, he needed a two-dimensional probability distribution that captured the relationship between errors in both directions. This led to the development of the bivariate normal distribution—the first widely-studied joint distribution.

The Mathematical Breakthrough

Francis Galton (1822-1911), studying heredity, discovered that parent height and child height followed a joint distribution with correlation. He invented the correlation coefficient and regression, both requiring joint distributions to formalize.

Karl Pearson (1857-1936) generalized these ideas, creating the theory of multivariate statistics. He showed that joint distributions are the mathematical language for describing how multiple quantities vary together.

From Single to Multiple Variables: The Conceptual Leap

Recall: Univariate Distributions

For a single discrete random variable $X$, the probability mass function (PMF) is:

This tells us: "What's the probability that $X$ takes the specific value $x$?" The PMF must satisfy:

• $p_X(x) \geq 0$ for all $x$ (probabilities are non-negative)
• $\sum_x p_X(x) = 1$ (total probability is 1)

For a continuous random variable $X$, the probability density function (PDF) is:

The PDF satisfies:

• $f_X(x) \geq 0$ for all $x$
• $\int_{-\infty}^{\infty} f_X(x)\,dx = 1$

The Leap to Two Variables

Now consider two random variables $X$ and $Y$. We want to describe the probability of observing the pair $(X, Y)$ taking specific values. This requires a joint distribution.

The key question changes from:

"What's the probability $X=x$?"

to:

"What's the probability $X=x$ and $Y=y$ simultaneously?"

This is a joint event: both conditions must hold at the same time.

Joint Probability Mass Function (PMF)

Mathematical Definition

For two discrete random variables $X$ and $Y$, the joint probability mass function is:

Let's unpack every symbol:

• $p_{X,Y}$: The joint PMF, a function of two variables
• $(x, y)$: A specific pair of values, one for $X$ and one for $Y$
• $P(X = x, Y = y)$: The probability that $X$ equals $x$ AND $Y$ equals $y$ in the same outcome

Axioms of Joint PMF

A valid joint PMF must satisfy two fundamental axioms:

1. Non-negativity:
   $$p_{X,Y}(x, y) \geq 0 \quad \text{for all } (x, y)$$

   Probabilities cannot be negative. Each entry in the joint probability table is ≥ 0.

2. Normalization (Total Probability = 1):
   $$\sum_x \sum_y p_{X,Y}(x, y) = 1$$

   Summing over all possible pairs $(x, y)$ must equal 1. This means we've accounted for all possible outcomes in the joint sample space.

Example: Rolling Two Dice

Let $X$ = outcome of first die, $Y$ = outcome of second die. Both take values in $\{1, 2, 3, 4, 5, 6\}$.

Since the dice are fair and independent:

for all $(x, y) \in \{1,\ldots,6\} \times \{1,\ldots,6\}$.

This is a uniform joint distribution over 36 outcomes. The joint PMF table is:

Verify: $36 \times \frac{1}{36} = 1$ ✓

Example: Dependent Variables

Now suppose $Y = X$ (the second die always shows the same as the first—perhaps they're glued together!). Then:

The joint PMF table becomes:

Verify: $6 \times \frac{1}{6} = 1$ ✓

This is an example of perfect positive dependence—knowing $X$ tells you exactly what $Y$ is.

Visualizing Joint PMF: Interactive Exploration

Joint PMFs can be visualized in multiple ways: probability tables (as above), heatmaps, or 3D bar charts. Let's explore interactively how different joint distributions look and behave.

Joint Probability Density Function (PDF)

Mathematical Definition

For two continuous random variables $X$ and $Y$, the joint probability density function is a function $f_{X,Y}(x, y)$ such that:

for any region $A \subseteq \mathbb{R}^2$.

Let's unpack this carefully:

• $f_{X,Y}(x, y)$: The joint PDF, a density (not a probability!) at point $(x, y)$
• $A$: A region in the $(x, y)$ plane (e.g., a rectangle, circle, or any 2D shape)
• $\iint_A \cdots dx\,dy$: A double integral over region $A$, which sums up density to get probability

Axioms of Joint PDF

1. Non-negativity:
   $$f_{X,Y}(x, y) \geq 0 \quad \text{for all } (x, y) \in \mathbb{R}^2$$
2. Normalization:
   $$\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f_{X,Y}(x, y)\,dx\,dy = 1$$

   The total "volume" under the joint PDF surface must equal 1.

Example: Uniform Distribution on a Square

Suppose $(X, Y)$ is uniformly distributed over the unit square $[0, 1] \times [0, 1]$. The joint PDF is:

Verify normalization:

The probability that $(X, Y)$ falls in a region $A$ is simply the area of $A$ (since density is constant at 1).

Example: Bivariate Normal Distribution

The bivariate normal distribution is the most important joint PDF in statistics. For $X \sim N(\mu_X, \sigma_X^2)$ and $Y \sim N(\mu_Y, \sigma_Y^2)$ with correlation $\rho$:

This looks intimidating! Let's decode it piece by piece:

• $\frac{1}{2\pi \sigma_X \sigma_Y \sqrt{1-\rho^2}}$: Normalization constant ensuring $\iint f = 1$
• $\exp(\cdots)$: Exponential decay from the center $(\mu_X, \mu_Y)$
• $\frac{(x-\mu_X)^2}{\sigma_X^2}$: Standardized squared distance in $X$ direction
• $\frac{(y-\mu_Y)^2}{\sigma_Y^2}$: Standardized squared distance in $Y$ direction
• $-\frac{2\rho(x-\mu_X)(y-\mu_Y)}{\sigma_X \sigma_Y}$: Correlation term that creates dependence! When $\rho \neq 0$, the contours are ellipses tilted at an angle.
• $\rho$: Correlation coefficient $\in [-1, 1]$. When $\rho = 0$, $X$ and $Y$ are independent.

Visualizing Joint PDF: Interactive 3D Surface

Joint PDFs are best understood through 3D surface plots and contour maps. The height of the surface at $(x, y)$ represents the density $f_{X,Y}(x, y)$. Let's explore how correlation affects the shape.

Independence vs. Dependence: The Factorization Test

Definition of Independence

Two random variables $X$ and $Y$ are independent if and only if:

This is called the factorization criterion. It says: "The joint distribution equals the product of marginal distributions."

Testing for Independence

To check if $X$ and $Y$ are independent:

1. Compute marginal PMFs/PDFs:
   $$p_X(x) = \sum_y p_{X,Y}(x, y) \quad \text{or} \quad f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x, y)\,dy$$
   $$p_Y(y) = \sum_x p_{X,Y}(x, y) \quad \text{or} \quad f_Y(y) = \int_{-\infty}^{\infty} f_{X,Y}(x, y)\,dx$$
2. Check if $p_{X,Y}(x, y) = p_X(x) \cdot p_Y(y)$ (or the continuous version) for all $(x, y)$
3. If any pair $(x, y)$ violates the factorization, $X$ and $Y$ are dependent

Real-World Examples: Where Joint Distributions Appear

1. Medical Diagnosis: Blood Pressure and Cholesterol

Let $X$ = systolic blood pressure (mmHg), $Y$ = LDL cholesterol (mg/dL). These are positively correlated in the population: people with high blood pressure often have high cholesterol.

The joint distribution $f_{X,Y}(x, y)$ might be a bivariate normal with $\rho \approx 0.6$. Knowing a patient has $X = 160$ mmHg (hypertension) increases the probability they have high cholesterol.

Why joint matters: Risk prediction models use $P(\text{heart attack} \mid X, Y)$, which requires the joint distribution to assess combined risk.

2. Image Processing: Neighboring Pixels

In a grayscale image, let $X$ = intensity of pixel (i, j), $Y$ = intensity of pixel (i, j+1) (horizontal neighbor).

These are highly correlated because natural images have smooth regions. The joint distribution $p_{X,Y}(x, y)$ has high mass along the diagonal $y \approx x$.

Why joint matters: Image compression (JPEG) exploits this dependence. Instead of encoding each pixel independently, we encode the difference (which has lower entropy).

3. Financial Portfolio: Stock Returns

Let $X$ = daily return of Stock A, $Y$ = daily return of Stock B. The joint distribution $f_{X,Y}(x, y)$ determines portfolio risk.

• If \rho > 0: Stocks move together (both crash or both rise). High risk when combined.
• If \rho < 0: Stocks hedge each other. When A drops, B often rises. Lower portfolio variance.
• If $\rho = 0$: Independent movements. Diversification reduces variance by $\sqrt{2}$.

Why joint matters: Modern portfolio theory (Markowitz) uses the full covariance matrix of asset returns, derived from joint distributions.

4. Sensor Fusion: Radar and Lidar

In autonomous vehicles, let $X$ = distance to obstacle measured by radar, $Y$ = distance measured by lidar. Both have measurement noise.

The joint distribution $f_{X,Y}(x, y)$ models both sensors together. If they're calibrated correctly, we expect $Y \approx X$ (high positive correlation).

Why joint matters: Kalman filters combine noisy measurements using the joint distribution to get an optimal estimate.

AI/ML Applications: Why Deep Learning Engineers Must Master Joint Distributions

"Neural networks are probability distribution estimators." — Modern ML perspective

1. Generative Models: Learning Joint Distributions of Pixels

Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) learn the joint distribution of all pixels in an image.

For a 28×28 grayscale image (like MNIST), there are 784 pixels. The joint distribution is:

where each $x_i \in \{0, \ldots, 255\}$. This is a 784-dimensional joint PMF with $256^{784} \approx 10^{1890}$ possible images!

GANs learn this distribution implicitly: the generator $G(z)$ maps noise $z$ to images that follow $p(x_1, \ldots, x_{784})$.

2. Multivariate Regression: Predicting Multiple Outputs

Suppose we want to predict both house price $Y_1$ and time on market $Y_2$ from features $X$ (square footage, location, etc.).

Instead of two separate regressions, we model the joint distribution:

This captures the correlation between price and time-on-market (expensive houses may sell slower). The loss function becomes:

which is the negative log-likelihood of the joint distribution.

3. Bayesian Neural Networks: Weights as Joint Distributions

In Bayesian deep learning, the neural network weights $W = (w_1, w_2, \ldots, w_n)$ are random variables with a joint prior distribution:

After observing data $D$, we compute the joint posterior:

This joint distribution quantifies uncertainty in the weights, enabling better calibration and out-of-distribution detection.

4. Markov Random Fields: Image Segmentation

In image segmentation, each pixel $i$ has a label $Y_i \in \{\text{background, object}\}$. The joint distribution over all labels is:

This is modeled as a Markov Random Field (MRF), where neighboring pixels have correlated labels (smoothness prior).

5. Reinforcement Learning: Joint Distribution of States and Actions

In RL, the trajectory is a sequence of (state, action) pairs. The joint distribution is:

The policy $\pi(a \mid s)$ and transition dynamics p(s' \mid s, a) define this joint distribution. Value functions estimate expected returns by marginalizing over possible futures.

Python Implementation: Hands-On Code

Working with Joint PMF

1import numpy as np
2import matplotlib.pyplot as plt
3from mpl_toolkits.mplot3d import Axes3D
4
5# Define joint PMF for discrete random variables X and Y
6# Example: Rolling two dice (X = first die, Y = second die)
7
8def joint_pmf_dice(x, y):
9    """
10    Joint PMF for two fair dice.
11    P(X=x, Y=y) = 1/36 for all valid (x,y) pairs
12    """
13    if 1 <= x <= 6 and 1 <= y <= 6:
14        return 1/36
15    return 0
16
17# Create grid of all possible outcomes
18X_vals = np.arange(1, 7)  # Die 1: {1, 2, 3, 4, 5, 6}
19Y_vals = np.arange(1, 7)  # Die 2: {1, 2, 3, 4, 5, 6}
20
21# Compute joint probability table
22joint_probs = np.zeros((len(X_vals), len(Y_vals)))
23for i, x in enumerate(X_vals):
24    for j, y in enumerate(Y_vals):
25        joint_probs[i, j] = joint_pmf_dice(x, y)
26
27print("Joint PMF Table:")
28print(joint_probs)
29
30# Verify it's a valid PMF (sums to 1)
31total_prob = np.sum(joint_probs)
32print(f"\\nTotal probability: {total_prob:.4f}")
33
34# Example: P(X + Y = 7) - probability sum equals 7
35sum_equals_7 = sum(joint_pmf_dice(x, 7-x) for x in range(1, 7))
36print(f"P(X + Y = 7) = {sum_equals_7:.4f}")

Working with Joint PDF

1import numpy as np
2import matplotlib.pyplot as plt
3from scipy.stats import multivariate_normal
4from mpl_toolkits.mplot3d import Axes3D
5
6# Define joint PDF for continuous random variables X and Y
7# Example: Bivariate Normal Distribution
8
9def joint_pdf_bivariate_normal(x, y, mu_x=0, mu_y=0,
10                                sigma_x=1, sigma_y=1, rho=0):
11    """
12    Joint PDF for bivariate normal distribution.
13
14    Parameters:
15    - (x, y): Point at which to evaluate PDF
16    - mu_x, mu_y: Means of X and Y
17    - sigma_x, sigma_y: Standard deviations
18    - rho: Correlation coefficient (-1 <= rho <= 1)
19
20    Returns:
21    - f(x,y): Joint probability density at (x, y)
22    """
23    # Construct covariance matrix
24    cov = [[sigma_x**2, rho * sigma_x * sigma_y],
25           [rho * sigma_x * sigma_y, sigma_y**2]]
26
27    # Create multivariate normal distribution
28    rv = multivariate_normal(mean=[mu_x, mu_y], cov=cov)
29
30    # Evaluate PDF at (x, y)
31    return rv.pdf([x, y])
32
33# Create grid for visualization
34x = np.linspace(-3, 3, 100)
35y = np.linspace(-3, 3, 100)
36X, Y = np.meshgrid(x, y)
37
38# Compute PDF at each grid point (with correlation rho=0.7)
39Z = np.zeros_like(X)
40for i in range(X.shape[0]):
41    for j in range(X.shape[1]):
42        Z[i, j] = joint_pdf_bivariate_normal(X[i, j], Y[i, j],
43                                             rho=0.7)
44
45# Verify integral over entire space ≈ 1
46dx = x[1] - x[0]
47dy = y[1] - y[0]
48total_prob = np.sum(Z) * dx * dy
49print(f"Integral of PDF (should be ≈ 1): {total_prob:.4f}")
50
51# Calculate P(X > 0, Y > 0) - first quadrant
52mask = (X > 0) & (Y > 0)
53prob_first_quadrant = np.sum(Z[mask]) * dx * dy
54print(f"P(X > 0, Y > 0) = {prob_first_quadrant:.4f}")