Variance and Standard Deviation

Learning Objectives

Historical Context

The Problem: Mean Isn't Enough

In the previous section, we learned that expected value tells us where the center of a distribution is. But consider these two scenarios:

Both investments have the same expected return of 10%. But would you treat them as equivalent? Clearly not! Investment B is much more risky—its returns swing wildly, while Investment A is stable.

The Core Insight: Expected value tells us where the distribution is centered. Variance tells us how spread out it is. Together, they give us a much richer picture of uncertainty.

What Does Variance Measure?

Think of variance as answering the question: "On average, how far are the values from the center?" (where "far" is measured in squared units).

Intuitive Picture

Interactive: Visualizing Spread

Use the slider below to see how variance affects the spread of a distribution. Notice how a larger variance makes the distribution wider and flatter.

Mathematical Definition

Breaking Down the Formula

So variance is literally: the expected value of the squared distance from the mean. It's a weighted average of "how far things are from the center," where farther deviations are penalized more heavily (because of squaring).

Why Do We Square the Deviations?

This is one of the most common questions students ask. Why not just average the deviations $(X - \mu)$ directly? Or use absolute values $|X - \mu|$?

Problem 1: Deviations Sum to Zero

If we try to compute $E[X - \mu]$, we get:

The positive and negative deviations always cancel out! This tells us nothing about spread.

Two Solutions

Why Squaring Wins

1. Differentiability: $x^2$ is differentiable everywhere, but $|x|$ has a kink at 0. This matters hugely for optimization (gradient descent!).
2. Mathematical tractability: For independent random variables, $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)$. This beautiful additivity property doesn't hold for MAD.
3. Connection to Euclidean geometry: Squared distance is the foundation of Euclidean space. This connects variance to least squares, PCA, and many ML algorithms.
4. Central Limit Theorem: The normal distribution emerges naturally from summing independent random variables, and it's parameterized by variance (not MAD).

Deep Insight: Variance "won" because of its mathematical elegance, not because it's the only valid measure of spread. In robust statistics, MAD is sometimes preferred because it's less sensitive to outliers.

Interactive: Why Squaring Works

Drag the data points and observe how positive and negative deviations cancel when not squared, but squaring captures the true spread.

The Computational Shortcut

There's an equivalent formula that's often easier to compute:

Proof

Discrete vs Continuous Formulas

Example: Variance of a Fair Die

For a fair 6-sided die, $P(X = k) = 1/6$ for $k = 1, 2, 3, 4, 5, 6$.

First, the mean: $\mu = \frac{1+2+3+4+5+6}{6} = 3.5$

Then the variance:

Variance of Common Distributions

Standard Deviation: Back to Original Units

There's one annoying thing about variance: if $X$ is measured in dollars, then $\text{Var}(X)$ is in dollars squared. That's not intuitive!

Standard deviation (SD or σ) is simply the square root of variance. It brings us back to the original units, making it much more interpretable.

Rule of Thumb: Standard deviation tells you roughly how far a "typical" value is from the mean. If σ = 10 and μ = 50, most values are within 10-20 units of 50.

Interactive: Understanding Units

See how variance and standard deviation relate, and how SD maintains interpretable units.

The 68-95-99.7 Rule

For normal distributions (bell curves), there's a beautiful pattern relating standard deviation to probability:

This means: if heights are normally distributed with μ = 170cm and σ = 10cm, then about 68% of people are between 160-180cm, 95% are between 150-190cm, and nearly everyone (99.7%) is between 140-200cm.

Properties of Variance

Understanding these properties is essential for working with variance in practice:

Interactive: Variance Properties

Adjust the scaling factor $a$ and shift $b$ to see how the variance changes. Notice that $b$ has no effect!

Population vs Sample Variance

In practice, we usually don't know the true population parameters. We estimate them from a sample. But there's a subtle issue...

Why n-1? (Bessel's Correction)

The $n-1$ in the denominator is called Bessel's correction. It makes the sample variance an unbiased estimator of the population variance.

The intuition:

• We use the sample mean $\bar{x}$ instead of the true mean $\mu$
• The sample mean is calculated from the same data, so it's already "optimized" to be close to the data points
• This causes us to underestimate the true spread
• Dividing by $n-1$ instead of $n$ corrects for this bias

Real-World Applications

Finance: Risk Management

Quality Control: Six Sigma

Physics: Measurement Uncertainty

Variance in Machine Learning

Variance appears everywhere in machine learning. Here are the key places:

1. Loss Functions

Mean Squared Error (MSE) is directly related to variance:

This is the sample variance of the residuals! We use squared error because it's differentiable (for gradient descent) and penalizes large errors heavily.

2. Weight Initialization

Xavier/Glorot and He initialization carefully control the variance of initial weights:

This prevents gradients from exploding or vanishing during training.

3. Batch Normalization

BatchNorm explicitly normalizes activations to have unit variance:

This stabilizes training by controlling the internal distribution of activations.

4. Regularization

L2 regularization (Ridge) reduces the variance of model predictions at the cost of introducing some bias. This is the bias-variance tradeoff in action!

The Bias-Variance Tradeoff

This is one of the most important concepts in machine learning. The expected prediction error can be decomposed as:

The Tradeoff: Simple models have high bias but low variance. Complex models have low bias but high variance. The goal is to find the sweet spot!

Interactive: Bias-Variance Tradeoff

Adjust model complexity to see how bias and variance change. Watch for the sweet spot where total error is minimized!

Common Pitfalls

Coefficient of Variation

Sometimes we want to compare variability across distributions with different scales. The coefficient of variation (CV) normalizes standard deviation by the mean:

Example: If stock A has μ = $100, σ = $20 (CV = 20%) and stock B has μ = $10, σ = $5 (CV = 50%), then stock B is relatively more variable even though its standard deviation is smaller.

Python Implementation

1import numpy as np
2
3# Sample data
4data = np.array([2, 4, 4, 4, 5, 5, 7, 9])
5
6# --- Method 1: From definition ---
7def variance_definition(x):
8    """Compute variance from the definition."""
9    mu = np.mean(x)
10    return np.mean((x - mu)**2)
11
12# --- Method 2: Computational formula ---
13def variance_computational(x):
14    """Compute variance using E[X²] - E[X]²."""
15    return np.mean(x**2) - np.mean(x)**2
16
17# --- NumPy functions ---
18# Population variance (divide by n)
19pop_var = np.var(data)
20
21# Sample variance (divide by n-1, Bessel's correction)
22sample_var = np.var(data, ddof=1)
23
24# Standard deviation
25pop_std = np.std(data)
26sample_std = np.std(data, ddof=1)
27
28print(f"Population variance: {pop_var:.4f}")
29print(f"Sample variance: {sample_var:.4f}")
30print(f"Population std: {pop_std:.4f}")
31print(f"Sample std: {sample_std:.4f}")
32
33# --- Verify the formulas give same result ---
34print(f"\nDefinitional formula: {variance_definition(data):.4f}")
35print(f"Computational formula: {variance_computational(data):.4f}")
36
37# --- For distributions ---
38from scipy import stats
39
40# Normal distribution: Var = σ²
41normal = stats.norm(loc=0, scale=2)  # μ=0, σ=2
42print(f"\nNormal(0,2) variance: {normal.var()}")  # Should be 4
43
44# Poisson: Var = λ
45poisson = stats.poisson(mu=5)  # λ=5
46print(f"Poisson(5) variance: {poisson.var()}")  # Should be 5
47
48# Bernoulli: Var = p(1-p)
49bernoulli = stats.bernoulli(p=0.3)
50print(f"Bernoulli(0.3) variance: {bernoulli.var()}")  # Should be 0.21

Test Your Understanding

Take this quiz to check your understanding of variance and standard deviation.

Summary

Final Thought: If expected value tells you where to aim, variance tells you how much you might miss by. Both are essential for understanding and quantifying uncertainty—the heart of statistics and machine learning.