Bias, Variance, and MSE

Learning Objectives

By the end of this section, you will be able to:

The Big Picture: How Good Is Your Estimator?

The Central Question: Given multiple ways to estimate a parameter, how do we decide which estimator is "best"?

In Section 01, we learned that an estimator $\hat{\theta}$ is a function that takes sample data and produces an estimate of the unknown true parameter $\theta_0$. But here's the crucial insight:

This randomness raises a fundamental question: How do we evaluate the quality of an estimator that produces different values each time?

We need metrics that capture two distinct types of error:

What Is Bias?

Intuitive Understanding of Bias

Imagine you own a bathroom scale that displays weights. Unknown to you, the scale has a manufacturing defect that makes it always show 2 kg more than the actual weight.

Bias is the systematic, consistent error that doesn't go away with more data. It represents where your estimator is "aiming" relative to the truth.

Here's why this matters: A biased estimator gives you a false sense of confidence. You might collect more and more data, seeing your estimates converge — but they're converging to the wrong value!

Mathematical Definition

Formally, the bias of an estimator $\hat{\theta}$ is the difference between its expected value and the true parameter:

An estimator is called unbiased if its bias is zero:

$\text{Unbiased: } E[\hat{\theta}] = \theta_0$

Bias Examples

What Is Variance?

Intuitive Understanding of Variance

Now let's consider a different scenario. Imagine two archers shooting at a target:

Variance measures the "spread" or "uncertainty" in your estimator. It tells you how much the estimate would change if you collected a different sample.

Unlike bias, variance can be reduced by collecting more data. With more observations, your estimate becomes more stable.

Equivalently, using the alternative formula:

$\text{Var}(\hat{\theta}) = E[\hat{\theta}^2] - (E[\hat{\theta}])^2$

Variance Examples

Sample Size Effects: Variance Shrinks, Bias Stays

A critical insight is that more data reduces variance but cannot fix bias. For the sample mean $\bar{X}$:

$\text{Var}(\bar{X}) = \frac{\sigma^2}{n} \xrightarrow{n \to \infty} 0$

The variance decreases proportionally to $1/n$, meaning the standard error (standard deviation of the estimator) decreases as $1/\sqrt{n}$. To halve your standard error, you need to quadruple your sample size!

What Is Mean Squared Error (MSE)?

Now we need a single number that captures the overall quality of an estimator, combining both bias and variance. This is the Mean Squared Error (MSE).

MSE measures the average squared distance between our estimate and the truth — the "typical" error magnitude, squared.

Why squared error? Several reasons:

1. Penalizes large errors more: An error of 10 contributes 100, but an error of 2 only contributes 4
2. Mathematical convenience: Differentiable and leads to elegant decomposition
3. Positive definite: No cancellation between over- and under-estimates

The Beautiful Decomposition: MSE = Bias² + Variance

Here's one of the most important results in all of statistics:

MSE as Risk: A Deeper View

Generalizing to Vector Estimators

So far we've focused on estimating a single parameter $\theta \in \mathbb{R}$. In practice, we often estimate vectors of parameters $\boldsymbol{\theta} \in \mathbb{R}^p$, such as all coefficients in a regression model.

Technical Assumptions & Connections

Worked Example: Bernoulli Mean Estimation

Let's apply bias, variance, and MSE to a concrete problem: estimating the probability of success $p$ from coin flips.

The Bias-Variance Tradeoff

Here's a profound insight that underlies all of statistics and machine learning:

In most real-world scenarios, reducing bias increases variance, and reducing variance increases bias.

This is the famous bias-variance tradeoff. It explains why:

ML Techniques and the Bias-Variance Tradeoff

Modern machine learning provides several powerful techniques to navigate the bias-variance tradeoff. Each technique works by shifting the balance between bias and variance in a controlled way.

Key Insight: Regularization techniques (Ridge, LASSO, Dropout, Early Stopping) all work by deliberately increasing bias to reduce variance by a larger amount, resulting in lower overall MSE. Ensemble methods like Bagging reduce variance through averaging. Cross-validation helps us find the sweet spot.

Real-World Applications

Python Implementation

Let's see these concepts in code. We'll compute bias, variance, and MSE for different estimators:

1import numpy as np
2import matplotlib.pyplot as plt
3
4def simulate_estimator_properties(true_mean=5.0, true_var=4.0, n_samples=30,
5                                   n_simulations=10000):
6    """
7    Simulate sampling to compute bias, variance, and MSE of the sample mean.
8    """
9    estimates = []
10
11    for _ in range(n_simulations):
12        # Draw a sample
13        sample = np.random.normal(true_mean, np.sqrt(true_var), n_samples)
14        # Compute sample mean (our estimator)
15        estimate = sample.mean()
16        estimates.append(estimate)
17
18    estimates = np.array(estimates)
19
20    # Compute properties
21    expected_value = estimates.mean()  # E[θ̂]
22    bias = expected_value - true_mean   # E[θ̂] - θ₀
23    variance = estimates.var()          # Var(θ̂)
24    mse = np.mean((estimates - true_mean)**2)  # E[(θ̂ - θ₀)²]
25
26    return {
27        'expected_value': expected_value,
28        'bias': bias,
29        'bias_squared': bias**2,
30        'variance': variance,
31        'mse': mse,
32        'mse_decomposition': bias**2 + variance,  # Should equal MSE!
33        'estimates': estimates
34    }
35
36# Run simulation
37results = simulate_estimator_properties()
38
39print("═══════════════════════════════════════════")
40print("  SAMPLE MEAN ESTIMATOR PROPERTIES")
41print("═══════════════════════════════════════════")
42print(f"  E[θ̂] (Expected Value):    {results['expected_value']:.4f}")
43print(f"  θ₀ (True Mean):           5.0000")
44print(f"  Bias:                     {results['bias']:.4f}")
45print(f"  Bias²:                    {results['bias_squared']:.6f}")
46print(f"  Variance:                 {results['variance']:.4f}")
47print(f"  MSE (direct):             {results['mse']:.4f}")
48print(f"  MSE (Bias² + Var):        {results['mse_decomposition']:.4f}")
49print("═══════════════════════════════════════════")
50print(f"  ✓ Verification: MSE ≈ Bias² + Var")
51print("═══════════════════════════════════════════")

1# Comparing two estimators: Sample Mean vs Biased Estimator
2def compare_estimators(true_mean=5.0, true_var=4.0, n_samples=10,
3                       n_simulations=10000):
4    """
5    Compare:
6    1. Sample mean (unbiased)
7    2. Shrinkage estimator: θ̂_shrink = 0.9 * X̄ + 0.5 (biased but lower variance)
8    """
9    unbiased_estimates = []
10    biased_estimates = []
11
12    for _ in range(n_simulations):
13        sample = np.random.normal(true_mean, np.sqrt(true_var), n_samples)
14
15        # Unbiased: sample mean
16        unbiased_est = sample.mean()
17
18        # Biased: shrinkage towards 5.0
19        # This adds bias but reduces variance
20        biased_est = 0.9 * sample.mean() + 0.5
21
22        unbiased_estimates.append(unbiased_est)
23        biased_estimates.append(biased_est)
24
25    unbiased = np.array(unbiased_estimates)
26    biased = np.array(biased_estimates)
27
28    # Compute metrics
29    def get_metrics(estimates, true_value):
30        exp_val = estimates.mean()
31        bias = exp_val - true_value
32        var = estimates.var()
33        mse = np.mean((estimates - true_value)**2)
34        return exp_val, bias, var, mse
35
36    u_exp, u_bias, u_var, u_mse = get_metrics(unbiased, true_mean)
37    b_exp, b_bias, b_var, b_mse = get_metrics(biased, true_mean)
38
39    print("
40" + "═"*60)
41    print("  ESTIMATOR COMPARISON: Unbiased vs Shrinkage")
42    print("═"*60)
43    print(f"  {'Metric':<20} {'Unbiased':>15} {'Shrinkage':>15}")
44    print("-"*60)
45    print(f"  {'E[θ̂]':<20} {u_exp:>15.4f} {b_exp:>15.4f}")
46    print(f"  {'Bias':<20} {u_bias:>15.4f} {b_bias:>15.4f}")
47    print(f"  {'Variance':<20} {u_var:>15.4f} {b_var:>15.4f}")
48    print(f"  {'MSE':<20} {u_mse:>15.4f} {b_mse:>15.4f}")
49    print("═"*60)
50
51    winner = "Unbiased" if u_mse < b_mse else "Shrinkage"
52    print(f"  Winner (Lower MSE): {winner}")
53    print("═"*60)
54
55compare_estimators()

Key Insights

Summary