Functions of Random Variables

Learning Objectives

By the end of this section, you will be able to:

1. Understand why transformations of random variables are fundamental to probability theory and machine learning
2. Apply the CDF method to find the distribution of any transformed random variable
3. Derive the PDF of a transformed variable using the change-of-variables formula with the Jacobian
4. Handle both monotonic and non-monotonic transformations correctly
5. Connect transformations to real-world applications in neural networks, normalizing flows, and data preprocessing
6. Implement transformation techniques in Python for simulation and analysis

The Big Picture: Transformations as the Heart of Statistics

"Given that we know the distribution of X, what is the distribution of g(X)?"— This fundamental question underlies almost every statistical technique.

Imagine you're a data scientist working with sensor measurements. Your sensor gives you raw voltage readings $X$, but you need to know the distribution of the squared signal power $Y = X^2$. Or perhaps you're building a neural network and need to understand how the ReLU activation $Y = \max(0, X)$ changes the distribution of your layer outputs.

This question is not just theoretical—it appears constantly in practice:

Historical Context

The study of transformed random variables has a rich history intertwined with the development of probability theory itself:

Why Transform Random Variables?

Before diving into the mathematics, let's understand why we need to transform random variables:

1. Modeling Real-World Relationships

Physical quantities are often related by nonlinear functions. If we know the distribution of one quantity, we need transformation techniques to find the distribution of the related quantity.

2. Simplifying Distributions for Analysis

Some distributions are easier to work with than others. Transformations can "normalize" skewed data or stabilize variance.

3. Generating Random Samples

The inverse transform method generates samples from any distribution by transforming uniform random numbers:

4. Understanding Neural Network Behavior

Every activation function in a neural network transforms the distribution of its inputs. Understanding these transformations is essential for:

• Proper weight initialization (Xavier/He initialization)
• Batch normalization design
• Understanding gradient flow
• Detecting and preventing dying neurons (ReLU)

Discrete Case: The PMF Method

Let's start with the simpler discrete case. If $X$ is a discrete random variable with PMF $p_X(x)$, and $Y = g(X)$, how do we find the PMF of $Y$?

Example: Squaring a Die Roll

Let $X$ be the result of a fair 6-sided die roll. What is the distribution of $Y = X^2$?

Since each $x$ maps to a unique $y = x^2$, the transformation is one-to-one, and probabilities transfer directly.

Example: Non-Injective Transformation

Now consider $Y = |X - 3.5|$ (distance from the center value 3.5):

Now multiple $x$ values map to the same $y$, so we sum their probabilities.

The CDF Method (Universal Approach)

The CDF method is the most general approach—it works for any transformation, monotonic or not, discrete or continuous.

Example: The Square Transformation

Let $X \\sim N(0, 1)$. Find the distribution of $Y = X^2$.

PDF Method: Monotonic Functions

When $g$ is monotonic (strictly increasing or strictly decreasing) and differentiable, we have a direct formula that avoids integration.

Why the Absolute Value of the Derivative?

The derivative term arises from conservation of probability. When we stretch or compress the x-axis, the height of the PDF must adjust to keep the total probability equal to 1.

Example: Linear Transformation

Let $X \\sim N(\\mu, \\sigma^2)$ and $Y = aX + b$ where $a \\neq 0$.

Example: Exponential Transformation

Let $X \\sim N(\\mu, \\sigma^2)$ and $Y = e^X$. Then $Y$ follows a log-normal distribution.

PDF Method: Non-Monotonic Functions

When $g$ is not monotonic, multiple values of $x$ can map to the same $y$. We must sum contributions from all "branches" of the inverse.

Example: The Absolute Value Transformation

Let $X \\sim N(0, 1)$ and $Y = |X|$.

Common Transformations Reference

Here are the most important transformations you'll encounter in practice:

AI/ML Applications

Understanding transformations of random variables is essential for modern machine learning. Here are the key applications:

1. Normalizing Flows for Generative Modeling

2. Reparameterization Trick in VAEs

3. Activation Function Analysis

4. Data Augmentation and Preprocessing

Python Implementation

Basic Transformation with the PDF Method

g(-2) = 4, g(2) = 4

1import numpy as np
2from scipy import stats
3import matplotlib.pyplot as plt
4
5# Define the transformation g(x) = x^2
6def g(x):
7    return x**2
8
9# For Y = X^2, the inverse has two branches
10def g_inv_positive(y):
11    return np.sqrt(y)
12
13def g_inv_negative(y):
14    return -np.sqrt(y)
15
16# The PDF of Y = X^2 where X ~ N(0, 1)
17def f_Y(y, f_X):
18    """Compute PDF of Y = X^2 using change of variables."""
19    y = np.atleast_1d(y)
20    result = np.zeros_like(y, dtype=float)
21
22    # Only valid for y > 0
23    valid = y > 0
24
25    # Jacobian: |d(sqrt(y))/dy| = 1/(2*sqrt(y))
26    jacobian = 1 / (2 * np.sqrt(y[valid]))
27
28    # Sum contributions from both branches
29    x_pos = np.sqrt(y[valid])
30    x_neg = -np.sqrt(y[valid])
31
32    result[valid] = (f_X(x_pos) + f_X(x_neg)) * jacobian
33    return result
34
35# Standard normal PDF
36f_X = stats.norm.pdf
37
38# Generate y values for plotting
39y_vals = np.linspace(0.001, 5, 500)
40
41# Compute transformed PDF
42pdf_Y = f_Y(y_vals, f_X)
43
44# Compare with chi-squared(1)
45chi2_pdf = stats.chi2.pdf(y_vals, df=1)
46
47print(f"Max difference from chi-squared: {np.max(np.abs(pdf_Y - chi2_pdf)):.2e}")

The CDF Method Implementation

1# CDF Method for Y = X^2 where X ~ N(0,1)
2def cdf_method_demo():
3    """Demonstrate the CDF method step by step."""
4
5    # Step 1: For Y = X^2, P(Y <= y) = P(X^2 <= y)
6    # Step 2: X^2 <= y means -sqrt(y) <= X <= sqrt(y)
7
8    # Step 3: Use the standard normal CDF
9    def F_Y(y):
10        """CDF of Y = X^2."""
11        y = np.atleast_1d(y)
12        result = np.zeros_like(y, dtype=float)
13        valid = y > 0
14        result[valid] = (stats.norm.cdf(np.sqrt(y[valid]))
15                        - stats.norm.cdf(-np.sqrt(y[valid])))
16        return result
17
18    # Step 4: Differentiate to get the PDF
19    def f_Y_numerical(y, h=1e-6):
20        """Numerical derivative of CDF."""
21        return (F_Y(y + h) - F_Y(y - h)) / (2 * h)
22
23    # Compare methods
24    y_test = np.array([0.5, 1.0, 2.0, 3.0])
25    print("y		CDF Method	Chi2 Truth")
26    print("-" * 45)
27    for y in y_test:
28        numerical = f_Y_numerical(y)
29        truth = stats.chi2.pdf(y, df=1)
30        print(f"{y:.1f}		{numerical:.6f}	{truth:.6f}")
31
32cdf_method_demo()

Practical: Simulating and Verifying Transformations

1import numpy as np
2from scipy import stats
3import matplotlib.pyplot as plt
4
5def verify_transformation(X_dist, g, Y_dist, n_samples=100000, title=""):
6    """
7    Verify a transformation by comparing:
8    1. Histogram of g(X) samples
9    2. Theoretical PDF of Y
10    """
11    # Generate samples from X
12    X_samples = X_dist.rvs(n_samples)
13
14    # Transform samples
15    Y_samples = g(X_samples)
16
17    # Filter out invalid values (e.g., inf, nan)
18    Y_samples = Y_samples[np.isfinite(Y_samples)]
19
20    # Plot
21    fig, ax = plt.subplots(figsize=(10, 6))
22
23    # Histogram of transformed samples
24    ax.hist(Y_samples, bins=100, density=True, alpha=0.7,
25            label='Empirical (simulated)', color='steelblue')
26
27    # Theoretical PDF
28    y_range = np.linspace(Y_samples.min(),
29                          np.percentile(Y_samples, 99), 500)
30    ax.plot(y_range, Y_dist.pdf(y_range), 'r-', lw=2,
31            label='Theoretical PDF')
32
33    ax.set_xlabel('y')
34    ax.set_ylabel('Density')
35    ax.set_title(title or 'Transformation Verification')
36    ax.legend()
37    plt.show()
38
39    # Kolmogorov-Smirnov test
40    ks_stat, p_value = stats.kstest(Y_samples, Y_dist.cdf)
41    print(f"KS test: statistic={ks_stat:.4f}, p-value={p_value:.4f}")
42
43# Example 1: X ~ N(0,1), Y = X^2 -> Chi-squared(1)
44verify_transformation(
45    X_dist=stats.norm(0, 1),
46    g=lambda x: x**2,
47    Y_dist=stats.chi2(df=1),
48    title=r'$X \sim N(0,1)$, $Y = X^2$ → $\chi^2(1)$'
49)
50
51# Example 2: X ~ N(0,1), Y = e^X -> Log-Normal(0, 1)
52verify_transformation(
53    X_dist=stats.norm(0, 1),
54    g=np.exp,
55    Y_dist=stats.lognorm(s=1, scale=1),
56    title=r'$X \sim N(0,1)$, $Y = e^X$ → LogNormal(0, 1)'
57)
58
59# Example 3: U ~ Uniform(0,1), Y = -ln(U) -> Exponential(1)
60verify_transformation(
61    X_dist=stats.uniform(0, 1),
62    g=lambda u: -np.log(u),
63    Y_dist=stats.expon(scale=1),
64    title=r'$U \sim Uniform(0,1)$, $Y = -\ln(U)$ → Exp(1)'
65)

Common Pitfalls

1# WRONG: Missing Jacobian!
2def f_Y_wrong(y):
3    return f_X(g_inv(y))  # Missing |dg_inv/dy|!
4
5# CORRECT: Include the Jacobian
6def f_Y_correct(y):
7    return f_X(g_inv(y)) * abs(d_g_inv(y))

1# For g(x) = x^2:
2# WRONG: Only one branch
3f_Y_wrong = f_X(sqrt(y)) * jacobian
4
5# CORRECT: Both branches
6f_Y_correct = f_X(sqrt(y)) * jacobian + f_X(-sqrt(y)) * jacobian

Test Your Understanding

Summary

Transformations of random variables are fundamental to probability theory and essential for modern machine learning. The key techniques are:

Key Formulas

Key Takeaways

1. The CDF method is universal—it works for any transformation, but may require solving inequalities and differentiating
2. The PDF method is more direct for differentiable transformations, but requires computing the Jacobian carefully
3. For non-monotonic functions, sum contributions from all branches of the inverse
4. Always account for domain changes—transformations often change the support of the distribution
5. The Jacobian factor ensures probability conservation— it's the "stretching factor" that adjusts heights when intervals are stretched or compressed
6. Normalizing flows in deep learning directly exploit the change-of-variables formula with tractable Jacobians
7. The inverse transform method generates samples from any distribution by transforming uniform random numbers

Coming Next: In the next section, we'll explore the Jacobian Transformation Method in detail, extending to multivariate transformations and learning how to compute Jacobian determinants efficiently.