Estimators and Their Properties

Learning Objectives

By the end of this section, you will be able to:

The Big Picture: Why Estimation Matters

Statistics is fundamentally about learning from data. We observe data, but we want to know about the underlying process that generated it.

The Detective Analogy

Think of estimation like being a detective. You arrive at a crime scene and find clues (your data). You can't rewind time to see exactly what happened (the true parameters), but you can use the clues to reconstruct what most likely occurred.

The better your detective methods (estimators), the closer your reconstruction gets to the truth. And just like in detective work, some methods are provably better than others.

A Concrete Example: The Coffee Shop Mystery

Imagine you're studying customer wait times at a coffee shop. You collect data: 2.3 min, 1.8 min, 4.1 min, 3.2 min...

The central question: What is the "true" average wait time for ALL customers (past, present, and future)?

Here's the key insight: that "true average" exists somewhere out there as a fixed number — maybe it's exactly 2.847 minutes. We'll never know it precisely, but we can get closer and closer with more data and smarter methods.

Why Should You Care? (The ML Connection)

Every time you train a machine learning model, you're doing estimation:

The Three Questions of Estimation

Every estimation problem comes down to three questions:

1. What should we estimate? — Defining the parameter(s) of interest
2. How should we estimate it? — Choosing an estimator (this chapter!)
3. How good is our estimate? — Quantifying uncertainty (next chapters)

Real-World Estimation: From Problems to Mathematical Models

Estimation isn't just abstract mathematics — it solves concrete problems every day. Click on each example below to see how real-world challenges map directly to the estimation framework we're learning.

The Parametric Framework

Before we can estimate anything, we need to set up our mathematical framework precisely. Let's decode the notation:

The Setup: Observation Space and Probability Families

$X \in \mathcal{X}, \quad X \sim P \in \mathcal{P}$

Let's break this down symbol by symbol:

The Parametric Assumption

In the parametric case, we assume the true distribution $P$ belongs to a specific family indexed by parameters:

$\mathcal{P} = \{P_{\boldsymbol{\theta}} : \boldsymbol{\theta} \in \Theta\}$

The key insight: Once we specify $\theta$, we completely determine the probability distribution. The estimation problem becomes: Which $\theta$ generated our data?

What Is an Estimator and Estimate?

An estimator $\hat{\theta}(X)$ is a machine (function) that turns data $X$ into guesses. An estimate $\hat{\theta}$ is the actual guess.

The estimator predicts the true parameter $\theta$ of the population using sample data $X$ from that population.

Formally, an estimator $\hat{\boldsymbol{\theta}}(X)$ is a function of the observation vector $X$ that produces an estimate of the unknown parameter $\theta$.

$\hat{\boldsymbol{\theta}} : \mathcal{X} \to \Theta$

This notation emphasizes three crucial points:

1. $\hat{\theta}$ is a function — It takes data as input and produces a parameter estimate as output
2. $\hat{\theta}$ depends only on $X$ — We can only use observable data, not the true (unknown) $\theta$
3. $\hat{\theta}$ lives in $\Theta$ — The estimate should be a valid parameter value

Estimator vs Estimate

A subtle but important distinction:

The estimator $\hat{\theta}(X)$ is a random variable because $X$ is random. The estimate $\hat{\theta}(x)$ is a specific number computed from realized data $x$.

Contrast Functions: Measuring Discrepancy

How do we find a "good" estimator? The key idea is to define a contrast function that measures how "far" any candidate parameter is from the truth.

Definition of Contrast Function

$\rho : \mathcal{X} \times \Theta \to \mathbb{R}$

A contrast function $\rho$ (Greek letter "rho") is a function that takes:

• An observation $X$ from the sample space $\mathcal{X}$
• A candidate parameter $\theta$ from the parameter space $\Theta$

And produces a real number measuring the "discrepancy" between $\theta$ and the truth.

Wait — How Can We Compare Data and Parameters?

You might be confused: $X$ is data (numbers we observed), and $\theta$ is a parameter (an abstract quantity describing a distribution). These are completely different types of objects! How can a function take both and produce a meaningful "discrepancy"?

Here's how it works:

1. The parameter $\theta$ defines a probability model — it tells us what the data should look like if $\theta$ were true
2. The data $X$ is what we actually observed — the real numbers we collected
3. The function $\rho$ measures the "fit" — how well does what we saw match what we'd expect if $\theta$ were true?

Concrete Examples

Let's make this crystal clear with examples. Click each to explore:

The Key Requirement

For $\rho$ to be a useful contrast function, we need a special property. Define the population discrepancy:

$D(\theta_0, \theta) \equiv E_{\theta_0}[\rho(X, \theta)]$

This measures the average discrepancy when the true parameter is $\theta_0$ and we're evaluating at $\theta$.

This requirement ensures that if we knew the truth ($\theta_0$), the contrast function would correctly identify it as the best choice.

The Discrepancy Function $D(\theta_0, \theta)$

Let's understand the discrepancy function more deeply:

For discrete distributions, the integral becomes a sum:

$D(\theta_0, \theta) = \sum_{x \in \mathcal{X}} \rho(x, \theta) \, P_{\theta_0}(X = x)$

1# Population discrepancy (ideal, but impossible):
2# D(θ₀, θ) = E_{x~P_θ₀}[ρ(x, θ)]
3
4# Empirical risk (what we actually compute):
5loss = 0
6for x, y in training_data:
7    loss += criterion(model(x), y)  # ρ(x, θ)
8loss = loss / len(training_data)    # (1/n) Σ ρ(xᵢ, θ)
9
10# Minimize!
11loss.backward()
12optimizer.step()

Interactive Geometric Visualization

Let's visualize this "bowl" concept interactively! Move the sliders to see how the discrepancy function behaves as you explore different parameter values.

Minimum Contrast Estimates

Since we can't compute $D(\theta_0, \theta)$ (we don't know $\theta_0$), we need a clever workaround. Here's the key insight:

Since $\rho(X, \theta)$ is an unbiased estimate of $D(\theta_0, \theta)$, we can minimize $\rho$ instead of $D$!

The Minimum Contrast Estimator

Define the minimum contrast estimate as:

$\hat{\theta}(X) = \arg\min_{\theta \in \Theta} \rho(X, \theta)$

In words: $\hat{\theta}(X)$ is the parameter value that minimizes the contrast function for the observed data $X$.

Why This Works

The fundamental idea is beautifully simple:

1. If we could compute $D(\theta_0, \theta)$ and minimize it, we'd find $\theta_0$ (the truth)
2. We can't compute $D$ directly ($\theta_0$ is unknown)
3. But $\rho(X, \theta)$ is an "unbiased estimate" of $D$ in a weak sense
4. So minimizing $\rho(X, \theta)$ should give us something close to $\theta_0$

Estimating Equations

Now we introduce a powerful technique: instead of directly minimizing the contrast, we use calculus to find where the derivative equals zero.

The Gradient Condition

Suppose $\Theta$ is Euclidean ($\Theta \subset \mathbb{R}^d$), $\theta_0$ is an interior point, and $D$ is smooth. Then at the minimum:

$\nabla_{\theta} D(\theta_0, \theta) \Big|_{\theta=\theta_0} = 0$

where $\nabla_{\theta}$ denotes the gradient (vector of partial derivatives):

$\nabla_{\theta} = \left(\frac{\partial}{\partial \theta_1}, \ldots, \frac{\partial}{\partial \theta_d}\right)$

The Estimating Equation

Since we can't evaluate $D$ directly, we use the sample version. The estimating equation is:

$\nabla_{\theta}\rho(X, \hat{\theta}) = 0$

In words: Find $\hat{\theta}$ such that the gradient of the contrast function (evaluated at $\hat{\theta}$) equals zero.

General Estimating Equations

There's an even more general framework that doesn't require starting from a contrast function. We directly specify estimating functions.

The $\Psi$-Function Approach

"More generally, suppose we are given a function $\Psi : \mathcal{X} \times \mathbb{R}^d \to \mathbb{R}^d$..."

Formally, $\Psi$ is a vector-valued function that takes data and a parameter, and outputs an "error vector":

$\Psi : \mathcal{X} \times \mathbb{R}^d \to \mathbb{R}^d$

where $\Psi \equiv (\psi_1, \ldots, \psi_d)^T$ — each component $\psi_j$ checks a different aspect of the parameter.

Population Condition

Define the population version:

$V(\theta_0, \theta) = E_{\theta_0}[\Psi(X, \theta)]$

We require that $V(\theta_0, \theta) = 0$ has $\theta_0$ as its unique solution.

This means: when the true parameter is $\theta_0$, the expected value of $\Psi(X, \theta)$ is zero only when $\theta = \theta_0$.

The Estimating Equation Estimate

We define $\hat{\theta}$ as the solution to:

$\Psi(X, \hat{\theta}) = 0$

This is $d$ equations in $d$ unknowns — one equation for each component of $\theta$.

Intuitive Examples

Let's make these abstract concepts concrete with familiar examples.

Example 1: Method of Moments

Setup: Estimate the mean $\mu$ from data $X_1, \ldots, X_n$.

Estimating function: $\psi(X_i, \mu) = X_i - \mu$

Estimating equation: $\sum_{i=1}^n (X_i - \hat{\mu}) = 0$

Solution: $\hat{\mu} = \bar{X} = \frac{1}{n}\sum_{i=1}^n X_i$

The sample mean! This is the simplest estimating equation estimate.

Example 2: Maximum Likelihood (Preview)

Setup: Observations with density $f(x; \theta)$.

Contrast function (negative log-likelihood):

$\rho(X, \theta) = -\sum_{i=1}^n \log f(X_i; \theta)$

Estimating equation (score equation):

$\sum_{i=1}^n \frac{\partial \log f(X_i; \theta)}{\partial \theta}\Big|_{\theta=\hat{\theta}} = 0$

This is the famous score equation — the foundation of Maximum Likelihood Estimation (covered in detail in Chapter 12).

Example 3: Least Squares

Setup: Linear regression $Y = X\beta + \varepsilon$.

Contrast function (sum of squared errors):

$\rho(Y, \beta) = \sum_{i=1}^n (Y_i - X_i^T\beta)^2$

Estimating equation (normal equations):

$X^T(Y - X\hat{\beta}) = 0$

Solution: $\hat{\beta} = (X^TX)^{-1}X^TY$

Complete Symbol Glossary

Here's a comprehensive reference for all the symbols in this section:

Python Implementation

Implementing a General Estimating Equation Solver

1import numpy as np
2from scipy.optimize import fsolve, minimize
3from typing import Callable
4
5def solve_estimating_equation(
6    psi: Callable[[np.ndarray, np.ndarray], np.ndarray],
7    X: np.ndarray,
8    theta_init: np.ndarray
9) -> np.ndarray:
10    """
11    Solve the estimating equation Psi(X, theta) = 0.
12
13    Parameters:
14    -----------
15    psi : Callable
16        Estimating function Psi(X, theta) -> R^d
17    X : np.ndarray
18        Observed data
19    theta_init : np.ndarray
20        Initial guess for theta
21
22    Returns:
23    --------
24    np.ndarray : Solution theta_hat
25    """
26    # Define the equation to solve: sum over observations
27    def equation(theta: np.ndarray) -> np.ndarray:
28        return np.sum([psi(x_i, theta) for x_i in X], axis=0)
29
30    # Solve using scipy's fsolve
31    theta_hat = fsolve(equation, theta_init)
32    return theta_hat
33
34
35# Example 1: Method of Moments for Normal(mu, sigma^2)
36def psi_normal(x: float, theta: np.ndarray) -> np.ndarray:
37    """
38    Estimating function for Normal(mu, sigma^2).
39    theta = (mu, sigma^2)
40    """
41    mu, sigma2 = theta
42    return np.array([
43        x - mu,                    # For mu: E[X - mu] = 0
44        (x - mu)**2 - sigma2       # For sigma^2: E[(X-mu)^2 - sigma^2] = 0
45    ])
46
47
48# Generate data from Normal(5, 4)
49np.random.seed(42)
50X = np.random.normal(loc=5, scale=2, size=100)
51
52# Solve
53theta_hat = solve_estimating_equation(psi_normal, X, theta_init=np.array([0.0, 1.0]))
54print(f"True theta = (5, 4)")
55print(f"Estimated theta_hat = ({theta_hat[0]:.4f}, {theta_hat[1]:.4f})")

Minimum Contrast Estimation

1import numpy as np
2from scipy.optimize import minimize
3from typing import Callable
4
5def minimum_contrast_estimate(
6    rho: Callable[[np.ndarray, np.ndarray], float],
7    X: np.ndarray,
8    theta_init: np.ndarray,
9    bounds: tuple = None
10) -> np.ndarray:
11    """
12    Find theta that minimizes the contrast function rho(X, theta).
13
14    Parameters:
15    -----------
16    rho : Callable
17        Contrast function rho(X, theta) -> R
18    X : np.ndarray
19        Observed data
20    theta_init : np.ndarray
21        Initial guess
22    bounds : tuple, optional
23        Parameter bounds
24
25    Returns:
26    --------
27    np.ndarray : Minimum contrast estimate theta_hat
28    """
29    def objective(theta: np.ndarray) -> float:
30        return rho(X, theta)
31
32    result = minimize(objective, theta_init, bounds=bounds, method='L-BFGS-B')
33    return result.x
34
35
36# Example: Least Squares Regression
37def least_squares_contrast(X: np.ndarray, beta: np.ndarray) -> float:
38    """
39    Contrast function: sum of squared residuals.
40    X has columns [1, x1, x2, ..., y] (last column is response)
41    """
42    design = X[:, :-1]  # Features
43    y = X[:, -1]        # Response
44    y_pred = design @ beta
45    return np.sum((y - y_pred)**2)
46
47
48# Generate linear regression data: y = 2 + 3*x + noise
49np.random.seed(42)
50n = 100
51x = np.random.uniform(0, 10, n)
52y = 2 + 3*x + np.random.normal(0, 1, n)
53X = np.column_stack([np.ones(n), x, y])  # [1, x, y]
54
55# Estimate
56beta_hat = minimum_contrast_estimate(
57    least_squares_contrast, X,
58    theta_init=np.array([0.0, 0.0])
59)
60print(f"True beta = (2, 3)")
61print(f"Estimated beta_hat = ({beta_hat[0]:.4f}, {beta_hat[1]:.4f})")

Visualizing the Discrepancy Function

1import numpy as np
2import matplotlib.pyplot as plt
3from scipy import stats
4
5# Setup: Estimate mean mu of Normal(mu, 1)
6true_mu = 3.0
7n = 50
8
9# Generate data
10np.random.seed(42)
11X = np.random.normal(loc=true_mu, scale=1, size=n)
12
13# Define contrast function (negative log-likelihood per observation)
14def contrast(x_i, mu):
15    """rho(x, mu) = (x - mu)^2 / 2 (proportional to negative log-likelihood)"""
16    return (x_i - mu)**2 / 2
17
18# Compute sample contrast for range of mu values
19mu_range = np.linspace(0, 6, 200)
20sample_contrast = [np.mean([contrast(x_i, mu) for x_i in X])
21                   for mu in mu_range]
22
23# The true discrepancy D(mu_0, mu) = E[(X - mu)^2/2] = ((mu - mu_0)^2 + 1)/2
24true_discrepancy = ((mu_range - true_mu)**2 + 1) / 2
25
26# Plot
27plt.figure(figsize=(10, 6))
28plt.plot(mu_range, sample_contrast, 'b-', linewidth=2,
29         label=r'Sample contrast $\bar{\rho}(X, \mu)$')
30plt.plot(mu_range, true_discrepancy, 'r--', linewidth=2,
31         label=r'True discrepancy $D(\mu_0, \mu)$')
32plt.axvline(true_mu, color='g', linestyle=':', linewidth=2,
33            label=f'True $\mu_0$ = {true_mu}')
34plt.axvline(np.mean(X), color='purple', linestyle=':', linewidth=2,
35            label=f'Estimate $\hat{{\mu}}$ = {np.mean(X):.3f}')
36
37plt.xlabel(r'$\mu$', fontsize=14)
38plt.ylabel('Contrast / Discrepancy', fontsize=14)
39plt.title('Sample Contrast vs Population Discrepancy', fontsize=14)
40plt.legend(fontsize=12)
41plt.grid(True, alpha=0.3)
42plt.tight_layout()
43plt.savefig('discrepancy_visualization.png', dpi=150)
44plt.show()
45
46# The minimum of sample contrast gives our estimate
47mu_hat = mu_range[np.argmin(sample_contrast)]
48print(f"True mu = {true_mu}")
49print(f"Sample mean = {np.mean(X):.4f}")
50print(f"Minimizer of sample contrast = {mu_hat:.4f}")

Key Insights

The Estimation Recipe

Why This Framework Is Powerful

• Unifying: Maximum Likelihood, Least Squares, Method of Moments are all special cases
• Flexible: Can handle complex models by choosing appropriate contrast functions
• Analyzable: Properties of $\hat{\theta}$ (bias, variance, consistency) follow from properties of $\rho$
• Computationally tractable: Estimating equations often easier to solve than direct optimization

Summary

This section introduced the foundational framework for estimation theory. Here are the key takeaways:

Core Concepts

The Big Ideas

1. Estimation is about finding the parameter that generated our data — we use observable quantities to infer unobservable truths
2. Contrast functions formalize "goodness of fit" — smaller contrast means better compatibility with data
3. The key requirement is identification — $D(\theta_0, \theta)$ must be uniquely minimized at $\theta_0$
4. Estimating equations are often easier to solve — differentiation turns optimization into root-finding

Coming Next: In the next section, we'll explore the key properties of estimators: bias, variance, and mean squared error (MSE). These concepts help us evaluate how "good" an estimator is and choose between competing estimation strategies.