Sufficiency and Minimal Sufficiency

Learning Objectives

By the end of this section, you will be able to:

Historical Context

"We may say that a statistic is sufficient for the purposes of estimation if it contains all the information which the sample contains of the value of the parameter."

— R.A. Fisher, 1922

Pitfalls and Caveats

Ancillarity & Completeness Bridge

Basu's Theorem

If T(X) is complete and sufficient for θ and A(X) is ancillary (its distribution does not depend on θ), then T and A are independent. This splits the data into an "information-carrying" part and a "noise-only" part.

Completeness and UMVUE

Completeness guarantees uniqueness of unbiased estimators: by Lehmann-Scheffé, the UMVUE is any unbiased function of a complete sufficient statistic.

• Bernoulli(p): T = ΣX_i is complete and sufficient; X̄ is the UMVUE for p.
• Poisson(λ): T = ΣX_i is complete and sufficient; X̄ is the UMVUE for λ.

We'll lean on completeness and ancillarity in the next section to build best unbiased estimators and tests.

The Big Picture: Data Compression Without Loss

Sufficiency asks: Can we summarize our data without losing any information about the parameter? It's the statistical equivalent of lossless compression.

In previous sections, we learned about estimator quality (bias, variance, MSE, consistency, efficiency). Now we ask a more fundamental question:

The Tax Return Analogy

Think about filing taxes:

The IRS doesn't need to see every receipt — the totals are sufficient. Similarly, for many statistical problems, a simple summary is sufficient for inference.

What Is Sufficiency?

Intuitive Understanding

A statistic T(X) is sufficient for parameter θ if, once you know T(X), the original data X tells you nothing more about θ.

Key Insight: Given T(X), the conditional distribution of X does not depend on θ. All the "information about θ" flows through T(X).

Formal Definition

What does this mean practically?

1. If two samples have the same T(X), they provide identical information about θ
2. Any estimator of θ based on X can be replaced by one based on T(X) alone
3. The MLE will always be a function of T(X)

The Factorization Theorem

The definition of sufficiency is conceptual. How do we actually find sufficient statistics? The Factorization Theorem provides the answer.

Why Factorization Works

The factorization theorem says:

Worked Examples

Proof Sketch

Animated Proof Walkthrough

Step through the proof interactively to see how each step builds on the previous:

Finding Sufficient Statistics

Exponential Family Shortcut

For distributions in the exponential family, finding sufficient statistics is automatic!

Additional Distribution Examples

Beyond the common exponential family members, here are more distributions with their sufficient statistics:

Pitman-Koopman-Darmois

Under mild regularity (iid samples, support not depending on θ), only exponential-family models admit a fixed-dimensional sufficient statistic as n grows. This is the Pitman-Koopman-Darmois theorem: if T(Xⁿ) has dimension not growing with n, the density must belong to an exponential family.

Regularity Conditions

The factorization theorem and related results require certain regularity conditions. Understanding when these fail helps avoid misapplying sufficiency.

Canonical GLM Examples

For canonical generalized linear models with fixed design matrix X, the sufficient statistics are theaggregated score terms:

• Logistic regression: Σ x_i y_i (vector) is sufficient for β.
• Poisson regression: Σ x_i y_i is sufficient for β.
• Gaussian GLM (known σ²): Σ x_i y_i, Σ x_ix_i^T summarize all β-information.

This is why distributed/mini-batch training can communicate gradients (scores) instead of raw data while retaining all likelihood information about the coefficients.

Minimal Sufficiency

Intuitive Understanding: Maximum Compression

Many statistics can be sufficient. The original data X is always (trivially) sufficient. But we want the most compressed sufficient statistic — one with no redundancy.

A minimal sufficient statistic is a sufficient statistic that is a function of every other sufficient statistic. It's the "coarsest" summary that still captures all information about θ.

Formal Definition

Discrete Minimality Example

For Bernoulli(p) data, the likelihood ratio method becomes explicit. For two samples x and y:

$\frac{f(x; p)}{f(y; p)} = p^{\sum x_i - \sum y_i} (1-p)^{n - \sum x_i - (n - \sum y_i)}$

This ratio is free of p if and only if Σx_i = Σy_i. Therefore, T(X) = ΣX_i is minimal sufficient for p. The same logic works for Poisson(λ): equality of sums defines the equivalence classes, so the total count is minimal sufficient.

Finding Minimal Sufficient Statistics

There's a powerful technique for finding minimal sufficient statistics:

Rao-Blackwell Improvement

The Rao-Blackwell Theorem shows that conditioning any estimator on a sufficient statistic can only reduce (or maintain) its variance. This provides a systematic way to improve estimators.

The Improvement Process:

1. Start with any unbiased estimator δ(X)
2. Find a sufficient statistic T(X)
3. Compute δ*(X) = E[δ(X) | T(X)]
4. The new estimator δ* has smaller (or equal) variance and is still unbiased

Likelihood Surface Visualization

One of the most powerful ways to understand sufficiency is to see that theentire likelihood function can be reconstructed from just the sufficient statistic. Two datasets with the same T(X) produce identical likelihood curves.

Sufficiency and MLE

There's a beautiful connection between sufficiency and Maximum Likelihood Estimation:

Common Mistakes

Here are frequent errors students and practitioners make with sufficiency. Test yourself by identifying the error before reading the explanation.

Bayesian Perspective

Sufficiency has a beautiful interpretation in Bayesian inference: the posterior depends on the data only through the sufficient statistic.

Why this works: By Bayes' theorem:

$p(\theta \mid X) \propto p(X \mid \theta) \cdot p(\theta) = g(T(X), \theta) \cdot h(X) \cdot p(\theta)$

Since h(X) doesn't depend on θ, it factors out and only g(T(X), θ) matters for the posterior.

Connection to Conjugate Priors

The exponential family connection explains why conjugate priors work so elegantly:

Information-Theoretic View

Information theory provides another lens for understanding sufficiency: a statistic is sufficient if and only if it preserves all mutual information with the parameter.

The Data Processing Inequality:

For any function T, we always have I(T(X); θ) ≤ I(X; θ) (processing data can only lose information). Sufficiency is the special case where equality holds!

Machine Learning Connections

The deepest insight connecting classical statistics to modern AI is this: sufficiency is the hidden soul of deep learning. Every neural network, every transformer, every generative model is secretly performing the same ancient operation that Fisher described a century ago — compressing infinite-dimensional reality into finite-dimensional representations that preserve all decision-relevant information.

The Ancient Question in a Modern World

Long before neural networks existed, statisticians asked a deceptively simple question:

The Central Mystery: How can we take an infinite, continuous, chaotic stream of reality — millions of pixels, billions of words, countless sensor readings — and compress it into a finite representation that loses nothing important?

The answer that emerged from both classical statistics and modern deep learning is remarkably unified: find the minimal sufficient statistic for your task.

The Story: From Raw Reality to a Decision Crystal

Imagine you're holding a 10-megapixel photograph — about 30 million numbers (RGB values). You need to answer one question: “Is this a cat or a dog?”

The question is: What is the right compression? Not too lossy (you might misclassify), not too large (inefficient, prone to overfitting). The answer: the minimal sufficient statistic for the classification task.

Two Worlds, One Principle

Modern AI Models as Sufficiency Machines

Every major deep learning architecture can be understood as a sufficiency machine — a system that compresses input into a representation that preserves all task-relevant information.

The Information Bottleneck = Minimal Sufficiency

The Information Bottleneck (IB) principle, introduced by Tishby et al., provides the formal connection between deep learning and minimal sufficiency.

This is exactly the definition of minimal sufficiency:

What this means for ML:

Tishby's Hypothesis: During training, networks first increase I(Z; Y) (fitting), then decrease I(X; Z) (compression). The result approximates the minimal sufficient statistic.

The Grand Unification

Here is the mapping between classical statistics and deep learning, revealing they are two languages for the same underlying mathematics:

Practical Applications

Real-World Applications

A/B Testing Case Study

A/B testing provides a perfect real-world example of sufficiency in action. Consider testing two versions of a website button to see which gets more clicks.

Why Sufficiency Matters for A/B Testing:

1. Privacy-Preserving Analysis: Share only (n_A, x_A, n_B, x_B) with analysts. No need to reveal individual user actions — the sufficient statistics contain all information needed for inference about p_A and p_B.
2. Efficient Storage: Store only 4 numbers instead of 20,000 individual records. For ongoing tests, just update the running totals.
3. Complete Inference: From these 4 numbers, we can compute:
   • Point estimates: p̂_A = x_A/n_A, p̂_B = x_B/n_B
   • Confidence intervals for p_A, p_B, and p_B - p_A
   • Chi-square test or Fisher's exact test
   • Bayesian posterior distributions

Python Implementation

Let's implement and verify sufficiency concepts in Python:

1import numpy as np
2from scipy import stats
3import matplotlib.pyplot as plt
4
5def demonstrate_sufficiency():
6    """Show that MLE depends only on sufficient statistics."""
7    np.random.seed(42)
8
9    # Generate Bernoulli data
10    true_p = 0.7
11    n = 100
12    data = np.random.binomial(1, true_p, n)
13
14    # Original data approach
15    mle_from_data = np.mean(data)
16
17    # Sufficient statistic approach
18    T = np.sum(data)  # Sufficient statistic
19    mle_from_T = T / n
20
21    print("Demonstrating Sufficiency for Bernoulli")
22    print("=" * 50)
23    print(f"Original data: {data[:20]}... (n={n})")
24    print(f"Sufficient statistic T = sum(X) = {T}")
25    print()
26    print(f"MLE from original data: {mle_from_data:.4f}")
27    print(f"MLE from sufficient T:  {mle_from_T:.4f}")
28    print(f"Same answer: {np.isclose(mle_from_data, mle_from_T)}")
29
30def verify_factorization_normal():
31    """Verify factorization for Normal distribution."""
32    np.random.seed(42)
33
34    mu_true, sigma_true = 5.0, 2.0
35    n = 50
36    data = np.random.normal(mu_true, sigma_true, n)
37
38    # Compute sufficient statistics
39    T1 = np.sum(data)       # Sum
40    T2 = np.sum(data**2)    # Sum of squares
41
42    # Derived sufficient statistics
43    x_bar = T1 / n
44    s_squared = (T2 - n * x_bar**2) / (n - 1)
45
46    print("
47Normal Distribution: Sufficient Statistics")
48    print("=" * 50)
49    print(f"Sample size: n = {n}")
50    print(f"
51Minimal Sufficient Statistics:")
52    print(f"  T1 = sum(X) = {T1:.4f}")
53    print(f"  T2 = sum(X^2) = {T2:.4f}")
54    print(f"
55Equivalent forms:")
56    print(f"  Sample mean = {x_bar:.4f} (true mu = {mu_true})")
57    print(f"  Sample var = {s_squared:.4f} (true sigma^2 = {sigma_true**2})")
58
59def compare_sufficient_vs_insufficient():
60    """Compare sufficient vs insufficient statistics for Uniform(0, theta)."""
61    np.random.seed(42)
62
63    theta_true = 10.0
64    n = 30
65    data = np.random.uniform(0, theta_true, n)
66
67    # Sufficient statistic: max(X)
68    T_sufficient = np.max(data)
69    mle_theta = T_sufficient  # MLE is max (biased, but consistent)
70
71    # Insufficient statistic: mean (loses information!)
72    insufficient_stat = np.mean(data)
73    naive_estimate = 2 * insufficient_stat  # Method of moments
74
75    print("
76Uniform(0, theta): Sufficient vs Insufficient")
77    print("=" * 50)
78    print(f"True theta: {theta_true}")
79    print(f"
80Sufficient statistic T = max(X) = {T_sufficient:.4f}")
81    print(f"MLE from sufficient: {mle_theta:.4f}")
82    print(f"
83Insufficient stat (mean) = {insufficient_stat:.4f}")
84    print(f"Naive estimate 2*mean = {naive_estimate:.4f}")
85    print(f"
86Note: Mean loses information about theta!")
87    print(f"Two samples with same mean but different max")
88    print(f"would give SAME mean-based estimate but DIFFERENT")
89    print(f"information about theta.")
90
91def demonstrate_minimal_sufficiency():
92    """Show that minimal sufficient = maximum compression."""
93    np.random.seed(42)
94
95    mu_true, sigma_true = 5.0, 2.0
96    n = 20
97    data = np.random.normal(mu_true, sigma_true, n)
98
99    # Various statistics and their dimensions
100    statistics = {
101        "Original data": (tuple(data), n),
102        "Order statistics": (tuple(sorted(data)), n),
103        "(sum, sum_sq)": ((np.sum(data), np.sum(data**2)), 2),
104        "(mean, var)": ((np.mean(data), np.var(data)), 2),
105    }
106
107    print("
108Minimal Sufficiency: Data Compression")
109    print("=" * 50)
110    print(f"Original sample size: n = {n}")
111    print(f"Parameters: mu (unknown), sigma^2 (unknown) -> 2 params")
112    print()
113
114    for name, (stat, dim) in statistics.items():
115        sufficient = "Yes" if dim <= 2 or name == "Order statistics" else "No"
116        minimal = "Yes" if dim == 2 else "No"
117        compression = f"{n} -> {dim}" if dim < n else "None"
118
119        print(f"{name:20s}: dim={dim:2d}, sufficient={sufficient:3s}, "
120              f"minimal={minimal:3s}, compression={compression}")
121
122def streaming_sufficient_statistics():
123    """Demonstrate streaming computation using sufficient statistics."""
124    np.random.seed(42)
125
126    print("
127Streaming Computation via Sufficient Statistics")
128    print("=" * 50)
129
130    # Initialize sufficient statistics
131    n = 0
132    sum_x = 0
133    sum_x2 = 0
134
135    # Stream data in batches
136    true_mu, true_sigma = 100, 15
137    batch_size = 100
138    n_batches = 10
139
140    print(f"Streaming {n_batches} batches of {batch_size} observations each")
141    print(f"True parameters: mu={true_mu}, sigma={true_sigma}")
142    print()
143
144    for batch in range(n_batches):
145        # New batch of data
146        new_data = np.random.normal(true_mu, true_sigma, batch_size)
147
148        # Update sufficient statistics (O(1) update, O(1) storage!)
149        n += len(new_data)
150        sum_x += np.sum(new_data)
151        sum_x2 += np.sum(new_data**2)
152
153        # Compute estimates from sufficient statistics
154        mean_estimate = sum_x / n
155        var_estimate = (sum_x2 - n * mean_estimate**2) / (n - 1)
156
157        print(f"After batch {batch+1}: n={n:4d}, "
158              f"mu_hat={mean_estimate:.2f}, sigma_hat={np.sqrt(var_estimate):.2f}")
159
160    print()
161    print(f"Final estimates: mu_hat={mean_estimate:.2f}, sigma_hat={np.sqrt(var_estimate):.2f}")
162    print(f"True values:     mu={true_mu}, sigma={true_sigma}")
163    print(f"
164Memory used: O(3) regardless of n = {n}!")
165
166# Run all demonstrations
167if __name__ == "__main__":
168    demonstrate_sufficiency()
169    verify_factorization_normal()
170    compare_sufficient_vs_insufficient()
171    demonstrate_minimal_sufficiency()
172    streaming_sufficient_statistics()

Interactive Quiz

Test your understanding of sufficiency concepts with this interactive quiz. Each question has immediate feedback with detailed explanations.

Practice Checks

Quick prompts to internalize the concepts:

1. Find T(X) for Beta(α, β) and Negative Binomial(r, p). Are they minimal?
2. Use the likelihood-ratio test to show minimality for Poisson(λ) (hint: compare sums).
3. Propose a statistic for Uniform(0, θ) that is not sufficient and explain why.
4. Compute the compression ratio n → dim(T) for Normal with known vs unknown variance.
5. Explain how you would add differential privacy noise to a sufficient statistic before sharing it.

Key Insights

Summary