Fisher Information and CRLB

Learning Objectives

By the end of this section, you will be able to:

Where You'll Apply This: Neural network optimization (natural gradient, K-FAC), uncertainty quantification, continual learning (EWC), reinforcement learning (TRPO, PPO), experimental design, sample size calculations, and model comparison.

The Big Picture: A Historical Journey

We've learned how to construct estimators (MoM, MLE), but how do we know if an estimator is any good? Is there a fundamental limit to how well we can estimate parameters? This section answers these profound questions.

The Cramér-Rao Lower Bound they discovered tells us something profound: there exists a fundamental limit to estimation precision that no amount of algorithmic cleverness can overcome. This is analogous to the Heisenberg uncertainty principle in physics — some uncertainties are irreducible.

The Score Function

Before defining Fisher Information, we need the score function — the gradient of the log-likelihood with respect to parameters.

Key Properties of the Score

1. Zero expectation at true parameter:
   $E_{\theta_0}[s(\theta_0)] = 0$
   The expected gradient is zero only at the truth — the likelihood is "balanced" there.
2. Variance equals Fisher Information:
   $\text{Var}_\theta[s(\theta)] = E_\theta[s(\theta)^2] = I(\theta)$
   The spread of the score tells us how "informative" the data is.

Fisher Information

Now we can define the central concept: Fisher Information — a measure of how much information the data X carries about the parameter θ.

The Curvature Intuition: Fisher Information measures the "sharpness" of the log-likelihood function. A sharply peaked likelihood means small changes in θ cause large changes in the likelihood — the data strongly "points to" a specific parameter value. That's high information!

Interactive: Fisher Information Explorer

Explore how Fisher Information changes across different distributions and parameter values. Notice how the curvature of the log-likelihood relates to the Fisher Information.

Fisher Information Matrix

When we have multiple parameters $\boldsymbol{\theta} = (\theta_1, \ldots, \theta_k)$, Fisher Information becomes a matrix:

The Cramér-Rao Lower Bound

Now for the main theorem — arguably one of the most important results in estimation theory.

Derivation of CRLB

Interactive: CRLB Visualizer

Watch how estimator variances approach the CRLB as sample size increases. The red dashed line is the fundamental limit — no unbiased estimator can go below it!

Efficiency of Estimators

An estimator that achieves the CRLB is called efficient. Efficiency measures how close an estimator comes to this theoretical optimum.

Interactive: Efficiency Comparison

Compare the efficiency of different estimators across distributions. Notice how some seemingly reasonable estimators can be surprisingly inefficient!

Worked Examples

Fisher Information in Deep Learning

Fisher Information is not just theoretical — it's at the heart of some of the most powerful techniques in modern deep learning. Let's explore these connections.

Natural Gradient Descent

Standard gradient descent treats all parameter directions equally in Euclidean space. But probability distributions don't live in Euclidean space — they live in a curved statistical manifold. Natural gradient respects this geometry.

Interactive: Natural Gradient Demo

Watch how natural gradient descent (blue) takes a more efficient path than standard gradient descent (red) by accounting for the geometry of the loss surface.

Elastic Weight Consolidation (EWC)

One of the most elegant applications of Fisher Information in deep learning is Elastic Weight Consolidation for continual learning.

Real-World Applications

Python Implementation

Here's how to compute Fisher Information and CRLB in Python:

1import numpy as np
2from scipy import stats
3from typing import Callable
4
5def fisher_info_bernoulli(p: float, n: int = 1) -> float:
6    """Fisher Information for Bernoulli(p).
7
8    I(p) = 1 / [p(1-p)] for single observation
9    """
10    if p <= 0 or p >= 1:
11        raise ValueError("p must be in (0, 1)")
12    return n / (p * (1 - p))
13
14def fisher_info_normal_mean(sigma: float, n: int = 1) -> float:
15    """Fisher Information for Normal mean (known variance).
16
17    I(μ) = 1/σ² for single observation
18    """
19    return n / (sigma ** 2)
20
21def fisher_info_exponential(lambda_: float, n: int = 1) -> float:
22    """Fisher Information for Exponential(λ).
23
24    I(λ) = 1/λ² for single observation
25    """
26    return n / (lambda_ ** 2)
27
28def fisher_info_poisson(lambda_: float, n: int = 1) -> float:
29    """Fisher Information for Poisson(λ).
30
31    I(λ) = 1/λ for single observation
32    """
33    return n / lambda_
34
35def numerical_fisher_info(
36    log_likelihood: Callable[[float, np.ndarray], float],
37    theta: float,
38    data: np.ndarray,
39    epsilon: float = 1e-5
40) -> float:
41    """Compute Fisher Information numerically via second derivative.
42
43    Uses central difference approximation.
44    """
45    ll_plus = log_likelihood(theta + epsilon, data)
46    ll_center = log_likelihood(theta, data)
47    ll_minus = log_likelihood(theta - epsilon, data)
48
49    # Negative expected second derivative
50    d2_ll = (ll_plus - 2 * ll_center + ll_minus) / (epsilon ** 2)
51    return -d2_ll  # Fisher Info is negative of this
52
53def cramér_rao_bound(fisher_info: float) -> float:
54    """Compute Cramér-Rao Lower Bound.
55
56    CRLB = 1 / I(θ)
57    """
58    return 1.0 / fisher_info
59
60def efficiency(estimator_variance: float, fisher_info: float) -> float:
61    """Compute efficiency of an estimator.
62
63    Efficiency = CRLB / Var(estimator) = 1 / [I(θ) × Var(estimator)]
64    """
65    crlb = cramér_rao_bound(fisher_info)
66    return crlb / estimator_variance
67
68# Example: Verify CRLB for Bernoulli
69if __name__ == "__main__":
70    p_true = 0.3
71    n = 100
72
73    # Fisher Information
74    I_p = fisher_info_bernoulli(p_true, n)
75    print(f"Fisher Information I(p={p_true}) with n={n}: {I_p:.4f}")
76
77    # CRLB
78    crlb = cramér_rao_bound(I_p)
79    print(f"Cramér-Rao Lower Bound: {crlb:.6f}")
80
81    # Theoretical variance of sample proportion
82    var_phat = p_true * (1 - p_true) / n
83    print(f"Var(p̂) = p(1-p)/n: {var_phat:.6f}")
84
85    # Verify efficiency (should be 1.0 = 100%)
86    eff = efficiency(var_phat, I_p)
87    print(f"Efficiency: {eff:.4f} ({eff*100:.1f}%)")
88
89    # Simulation verification
90    np.random.seed(42)
91    n_simulations = 10000
92    estimates = []
93
94    for _ in range(n_simulations):
95        sample = np.random.binomial(1, p_true, n)
96        estimates.append(np.mean(sample))
97
98    empirical_variance = np.var(estimates, ddof=1)
99    print(f"\nSimulated Var(p̂): {empirical_variance:.6f}")
100    print(f"Ratio to CRLB: {empirical_variance / crlb:.4f}")

Common Pitfalls

Knowledge Check

Test your understanding of Fisher Information and the Cramér-Rao Lower Bound.

Summary