Consistency and Efficiency

Learning Objectives

By the end of this section, you will be able to:

The Big Picture: Beyond Finite Samples

Bias and variance tell us about an estimator's behavior for a fixed sample size. Consistency and efficiency tell us about its behavior as we gather more data and how good it is compared to what's theoretically possible.

In the previous section, we learned that MSE = Bias² + Variance captures the total error of an estimator. But this raises important questions:

Historical Context: The Giants of Estimation Theory

What Is Consistency?

Intuitive Understanding: The GPS Analogy

Imagine you have a GPS device trying to locate your position. Each reading has some error due to signal interference. Consistency means that if you average enough readings, you'll eventually pinpoint your exact location.

Consistency is the guarantee that with enough data, you will find the truth. It's the fundamental requirement for any estimator we want to trust.

Mathematical Definition

An estimator $\hat{\theta}_n$ is consistent for parameter $\theta_0$ if:

$\hat{\theta}_n \xrightarrow{p} \theta_0 \quad \text{as } n \to \infty$

This is read as "$\hat{\theta}_n$ converges in probability to $\theta_0$". Formally:

$\forall \varepsilon > 0: \quad \lim_{n \to \infty} P(|\hat{\theta}_n - \theta_0| > \varepsilon) = 0$

Strong vs Weak Consistency

There are actually two types of consistency:

Connection to MSE

Here's a beautiful connection to what we learned in Section 02:

Why this works: By Chebyshev's inequality:

$P(|\hat{\theta}_n - \theta_0| > \varepsilon) \leq \frac{\text{MSE}(\hat{\theta}_n)}{\varepsilon^2}$

If MSE → 0, then this probability → 0 for any ε, which is exactly consistency!

Consistency Examples

What Is Efficiency?

Intuitive Understanding: The Fuel Economy Analogy

Imagine two cars that both get you from A to B (both are consistent). But one car uses 5 liters of fuel while the other uses 8 liters. The first car is more efficient — it achieves the same goal with less resource.

An efficient estimator extracts all the "information" in the data. No other unbiased estimator can do better (have lower variance).

Fisher Information: Measuring Information Content

Before we can talk about efficiency, we need to understand Fisher Information — the amount of information that data provides about the parameter.

Visualizing Fisher Information: The Geometry of Estimation

The best way to understand Fisher Information is to see it in action. The interactive demos below let you explore how the log-likelihood curvature relates to estimation precision.

Fisher Information for Common Distributions

For n independent observations: The total Fisher Information is:

$I_n(\theta) = n \cdot I(\theta)$

More data means more information — linearly!

The Cramér-Rao Lower Bound (CRLB)

Now comes one of the most important theorems in statistics:

Why is this profound?

1. It tells us the best possible precision achievable
2. We can check if our estimator reaches this limit
3. It shows that variance decreases like 1/n (at best)
4. The limit depends only on the distribution, not on our estimation method

Try It: CRLB Calculator

Efficient Estimators

An estimator that achieves the CRLB is called efficient:

$\text{Var}(\hat{\theta}) = \frac{1}{n \cdot I(\theta)}$

Relative Efficiency

To compare two unbiased estimators, we use relative efficiency:

$\text{RE}(\hat{\theta}_1, \hat{\theta}_2) = \frac{\text{Var}(\hat{\theta}_2)}{\text{Var}(\hat{\theta}_1)}$

If RE > 1, then θ̂₁ is more efficient (lower variance).

Asymptotic Efficiency

Many estimators aren't efficient for finite n but become efficient as n → ∞. This is asymptotic efficiency.

This is why MLE is so popular:

• Consistent — converges to truth
• Asymptotically normal — nice distribution
• Asymptotically efficient — minimum variance

The Robustness-Efficiency Tradeoff

The practical wisdom: Use efficient estimators when you trust your model and data quality. Use robust estimators when outliers or model misspecification are concerns. In machine learning, this tradeoff appears in many places:

• L2 loss (MSE) is efficient but sensitive to outliers
• L1 loss (MAE) is robust but less efficient
• Huber loss tries to get the best of both worlds

Consistency vs Efficiency: The Complete Picture

Let's summarize the relationship between consistency and efficiency with a 2×2 matrix:

Common Misconceptions

Let's clear up some frequent misunderstandings about consistency and efficiency:

Worked Example: Verifying Consistency and Efficiency

Let's work through a complete example, verifying both consistency and efficiency for the MLE of an exponential distribution.

Machine Learning Connections

The concepts of consistency and efficiency appear throughout machine learning, often under different names. Understanding these connections deepens your intuition.

Real-World Applications

Python Implementation

Let's implement and verify these concepts in Python:

1import numpy as np
2from scipy import stats
3import matplotlib.pyplot as plt
4
5def demonstrate_consistency():
6    """Show that sample mean is consistent for population mean."""
7    np.random.seed(42)
8
9    true_mu = 5.0
10    sigma = 2.0
11    sample_sizes = [10, 50, 100, 500, 1000, 5000]
12
13    results = []
14    for n in sample_sizes:
15        # Take many samples and compute means
16        estimates = [np.mean(np.random.normal(true_mu, sigma, n))
17                     for _ in range(1000)]
18
19        # Check how close they are to true value
20        mse = np.mean([(est - true_mu)**2 for est in estimates])
21        variance = np.var(estimates)
22
23        results.append({
24            'n': n,
25            'mean_estimate': np.mean(estimates),
26            'variance': variance,
27            'theoretical_var': sigma**2 / n,
28            'mse': mse
29        })
30        print(f"n={n:5d}: Mean={np.mean(estimates):.4f}, "
31              f"Var={variance:.6f}, MSE={mse:.6f}")
32
33    return results
34
35def compute_fisher_information_bernoulli(p, n=1):
36    """Fisher Information for Bernoulli(p)."""
37    return n / (p * (1 - p))
38
39def compute_cramer_rao_bound(fisher_info):
40    """CRLB: minimum variance for unbiased estimator."""
41    return 1 / fisher_info
42
43def compare_efficiency():
44    """Compare sample mean vs median for normal data."""
45    np.random.seed(42)
46
47    true_mu = 10.0
48    sigma = 3.0
49    n = 50
50    n_simulations = 10000
51
52    mean_estimates = []
53    median_estimates = []
54
55    for _ in range(n_simulations):
56        sample = np.random.normal(true_mu, sigma, n)
57        mean_estimates.append(np.mean(sample))
58        median_estimates.append(np.median(sample))
59
60    var_mean = np.var(mean_estimates)
61    var_median = np.var(median_estimates)
62
63    # Theoretical values
64    theoretical_var_mean = sigma**2 / n  # CRLB
65    theoretical_var_median = (np.pi / 2) * sigma**2 / n
66
67    relative_efficiency = var_mean / var_median
68
69    print("Efficiency Comparison: Mean vs Median (Normal Data)")
70    print("=" * 50)
71    print(f"Sample size: {n}")
72    print(f"Simulations: {n_simulations}")
73    print()
74    print(f"Sample Mean Variance:    {var_mean:.6f}")
75    print(f"  (Theoretical CRLB:     {theoretical_var_mean:.6f})")
76    print()
77    print(f"Sample Median Variance:  {var_median:.6f}")
78    print(f"  (Theoretical:          {theoretical_var_median:.6f})")
79    print()
80    print(f"Relative Efficiency (median/mean): {relative_efficiency:.4f}")
81    print(f"  (Theoretical: 2/pi = {2/np.pi:.4f})")
82
83    return var_mean, var_median, relative_efficiency
84
85def verify_mle_efficiency():
86    """Verify MLE achieves CRLB for Bernoulli."""
87    np.random.seed(42)
88
89    true_p = 0.3
90    n = 100
91    n_simulations = 10000
92
93    # Fisher Information and CRLB
94    fisher_info = compute_fisher_information_bernoulli(true_p, n)
95    crlb = compute_cramer_rao_bound(fisher_info)
96
97    # Simulate MLE (sample proportion)
98    mle_estimates = []
99    for _ in range(n_simulations):
100        sample = np.random.binomial(1, true_p, n)
101        mle_estimates.append(np.mean(sample))
102
103    var_mle = np.var(mle_estimates)
104
105    print("MLE Efficiency for Bernoulli")
106    print("=" * 40)
107    print(f"True p: {true_p}")
108    print(f"Sample size: {n}")
109    print(f"Fisher Information: {fisher_info:.4f}")
110    print(f"CRLB (minimum variance): {crlb:.6f}")
111    print(f"MLE Variance:            {var_mle:.6f}")
112    print(f"Achieves CRLB: {np.isclose(var_mle, crlb, rtol=0.1)}")
113
114    return fisher_info, crlb, var_mle
115
116# Run demonstrations
117if __name__ == "__main__":
118    print("DEMONSTRATION: Consistency")
119    print("=" * 50)
120    demonstrate_consistency()
121    print()
122
123    print("DEMONSTRATION: Efficiency Comparison")
124    print("=" * 50)
125    compare_efficiency()
126    print()
127
128    print("DEMONSTRATION: MLE Achieves CRLB")
129    print("=" * 50)
130    verify_mle_efficiency()

Practice Problems

Test your understanding with these practice problems. Try to solve them before revealing the solutions!

Key Insights

Summary