Bootstrap Confidence Intervals

Learning Objectives

By the end of this section, you will be able to:

Where You'll Apply This: Model evaluation uncertainty, ensemble methods (Random Forests, Bagging), A/B testing for complex metrics, cross-validation confidence intervals, uncertainty quantification in neural networks, and any situation where analytical formulas are unavailable or unreliable.

The Big Picture: The Resampling Revolution

Consider this problem: you've calculated the correlation between customer satisfaction scores and purchase frequency from a sample of 50 customers, getting $r = 0.42$. What's the confidence interval for this correlation?

Before 1979, answering this question required either:

• Complex mathematical derivations of the sampling distribution (Fisher's z-transformation for correlations)
• Assumptions about the underlying population distribution that may not hold
• Giving up and reporting just the point estimate

The bootstrap changed everything. Instead of deriving formulas, we let the computer do the work: resample from our data, calculate the statistic, repeat thousands of times, and use the resulting distribution to form a confidence interval. No formulas required. No distributional assumptions needed.

Historical Context

The bootstrap arrived at the perfect time. Computers were becoming powerful enough to perform thousands of resampling operations, making the method practical. Today, bootstrap is one of the most widely used statistical techniques in science, medicine, and machine learning.

The Core Intuition

The bootstrap rests on a profound but simple idea:

Think of it this way: your sample is the best information you have about the population. The empirical distribution—which puts probability 1/n on each observed value—is a reasonable approximation to the unknown population distribution. By resampling from this empirical distribution, we simulate the process of taking new samples from the population.

The Bootstrap Algorithm

Interactive: Bootstrap Distribution Builder

Experience the bootstrap in action. Watch as we repeatedly resample from the original data and build up the bootstrap distribution of the sample mean. Notice how some observations appear multiple times in each bootstrap sample (highlighted with counts).

Why Does Bootstrap Work?

At first glance, the bootstrap seems like magic—or worse, circular logic. How can sampling from our sample tell us anything we don't already know? The key insight is that we're not trying to learn new facts about the population; we're trying to understand the variability of our estimator.

Mathematical Foundation

Let $F$ be the unknown true distribution and $\hat{F}_n$ be the empirical distribution function (EDF) that puts mass 1/n at each observed value. The bootstrap works because:

This means that for large samples, $\hat{F}_n$ is an excellent approximation to $F$. Therefore, the sampling distribution of a statistic computed from $\hat{F}_n$ (via bootstrap) should be close to the true sampling distribution computed from $F$.

More precisely, for many statistics $\hat{\theta}$, the bootstrap distribution converges to the true sampling distribution:

Bootstrap CI Methods

Once we have the bootstrap distribution, there are several ways to construct a confidence interval. Each method has different properties and is appropriate in different situations.

Percentile Method

The simplest approach: use the $\alpha/2$ and $1-\alpha/2$ percentiles of the bootstrap distribution directly as the CI bounds.

• Pros: Simple, intuitive, transformation-invariant
• Cons: Can be biased if the bootstrap distribution is asymmetric

Basic (Reverse Percentile) Method

This method reflects the percentiles around the original estimate to correct for bias:

• Pros: Partially corrects for bias in the bootstrap distribution
• Cons: Not transformation-invariant, can give intervals outside valid range

BCa (Bias-Corrected and Accelerated) Method

The most sophisticated and generally recommended method. It adjusts for both:

1. Bias (z₀): How much the bootstrap distribution is shifted from the original estimate
2. Acceleration (a): How much the standard error changes with the parameter value (skewness)

Interactive: Methods Comparison

Compare how different bootstrap CI methods behave, especially with skewed data. Try adjusting the skewness parameter to see how the methods diverge when the sampling distribution is asymmetric.

Interactive: Coverage Simulation

The true test of a CI method is its coverage: does a 95% CI actually contain the true parameter about 95% of the time? This simulation draws many samples from a population with known mean, constructs bootstrap CIs for each, and checks the actual coverage rate.

When to Use Bootstrap

AI/ML Applications

Bootstrap is not just a statistical technique—it's deeply embedded in modern machine learning. Understanding bootstrap gives you insight into some of the most powerful ML methods.

Bagging and Ensemble Methods

Random Forests extend bagging by adding random feature selection at each split, further decorrelating the trees. The bootstrap is essential to this variance reduction.

Out-of-Bag (OOB) Error

Here's a beautiful property of bootstrap sampling: each bootstrap sample excludes about 36.8% of the original observations (on average). These "out-of-bag" points provide a natural validation set!

For each training observation, we can collect predictions from all trees where that observation was out-of-bag, then average them. This gives an honest estimate of model performance without needing a separate test set—extremely useful when data is limited.

Uncertainty Quantification

Python Implementation

Here's a complete implementation of bootstrap confidence intervals, including the BCa method. Click on any highlighted line to see a detailed explanation.

rng = np.random.default_rng(seed=42)

sample = [3, 1, 3, 2, 3] could result from [1, 2, 3] with replacement

statistic=lambda x: np.percentile(x, 75) for 75th percentile CI

1import numpy as np
2from scipy import stats
3
4rng = np.random.default_rng(seed=42)
5
6
7def bootstrap_ci(data, statistic=np.mean, B=2000, confidence=0.95, method='percentile'):
8    """
9    Compute bootstrap confidence interval for any statistic.
10
11    Parameters
12    ----------
13    data : array-like
14        Observed sample data
15    statistic : callable
16        Function to compute the statistic (default: np.mean)
17    B : int
18        Number of bootstrap samples
19    confidence : float
20        Confidence level
21    method : str
22        'percentile', 'basic', or 'bca'
23    """
24    data = np.asarray(data)
25    n = len(data)
26    theta_hat = statistic(data)
27
28    # Generate bootstrap distribution
29    bootstrap_stats = np.array([
30        statistic(rng.choice(data, size=n, replace=True))
31        for _ in range(B)
32    ])
33
34    alpha = 1 - confidence
35
36    if method == 'percentile':
37        lower = np.percentile(bootstrap_stats, 100 * alpha / 2)
38        upper = np.percentile(bootstrap_stats, 100 * (1 - alpha / 2))
39
40    elif method == 'basic':
41        lower = 2 * theta_hat - np.percentile(bootstrap_stats, 100 * (1 - alpha / 2))
42        upper = 2 * theta_hat - np.percentile(bootstrap_stats, 100 * alpha / 2)
43
44    elif method == 'bca':
45        # Bias correction
46        z0 = stats.norm.ppf(np.mean(bootstrap_stats < theta_hat))
47
48        # Acceleration (jackknife)
49        jackknife_stats = np.array([
50            statistic(np.delete(data, i))
51            for i in range(n)
52        ])
53        jack_mean = np.mean(jackknife_stats)
54        num = np.sum((jack_mean - jackknife_stats) ** 3)
55        denom = 6 * np.sum((jack_mean - jackknife_stats) ** 2) ** 1.5
56        a = num / denom if denom != 0 else 0
57
58        # Adjusted percentiles
59        z_alpha_low = stats.norm.ppf(alpha / 2)
60        z_alpha_high = stats.norm.ppf(1 - alpha / 2)
61
62        alpha1 = stats.norm.cdf(z0 + (z0 + z_alpha_low) / (1 - a * (z0 + z_alpha_low)))
63        alpha2 = stats.norm.cdf(z0 + (z0 + z_alpha_high) / (1 - a * (z0 + z_alpha_high)))
64
65        lower = np.percentile(bootstrap_stats, 100 * alpha1)
66        upper = np.percentile(bootstrap_stats, 100 * alpha2)
67
68    return {
69        'estimate': theta_hat,
70        'ci': (lower, upper),
71        'se': np.std(bootstrap_stats, ddof=1),
72        'bootstrap_distribution': bootstrap_stats
73    }
74
75
76# Example: CI for correlation coefficient
77np.random.seed(123)
78n = 50
79x = np.random.normal(0, 1, n)
80y = 0.5 * x + np.random.normal(0, 0.8, n)
81
82def correlation(data):
83    """Compute correlation (expects stacked [x, y] array)"""
84    return np.corrcoef(data[0], data[1])[0, 1]
85
86stacked_data = np.vstack([x, y])
87result = bootstrap_ci(stacked_data, statistic=correlation, method='bca')
88
89print(f"Correlation: {result['estimate']:.3f}")
90print(f"95% BCa CI: [{result['ci'][0]:.3f}, {result['ci'][1]:.3f}]")
91print(f"Bootstrap SE: {result['se']:.3f}")

Knowledge Check

Test your understanding of bootstrap methods with this comprehensive quiz. Pay attention to both the intuition and the technical details.

Summary

Looking Ahead: In the next section, we'll explore Credible Intervals—the Bayesian counterpart to confidence intervals. We'll see how Bayesian methods provide a different interpretation of uncertainty and when each approach is most appropriate.