Chi-Square Distribution

Learning Objectives

By the end of this section, you will be able to:

1. Understand fundamentally why the chi-square distribution arises naturally as the sum of squared standard normal random variables
2. Define the chi-square distribution $\chi^2_k$ and explain the meaning of degrees of freedom
3. Recognize chi-square as a special case of Gamma: $\chi^2_k = \text{Gamma}(k/2, 1/2)$
4. Apply chi-square tests for goodness-of-fit and independence testing
5. Use chi-square for variance estimation and confidence intervals
6. Implement chi-square feature selection in machine learning pipelines
7. Connect chi-square to the F-distribution and Student's t-distribution

Deep Intuition: The Surprise Meter

The chi-square distribution measures "how surprising" your data is compared to what you expected.

Imagine you roll a die 600 times. You expect about 100 of each face. But you get: 90, 105, 115, 95, 85, 110. Is this die fair, or is something fishy?

The chi-square distribution answers this question. It tells you: "Given random variation, how likely is it to see this much deviation from expectation?"

Why Squared?

The Historical Story

The chi-square distribution emerged from fundamental questions about data, errors, and testing scientific theories.

Friedrich Robert Helmert (1876)

The German mathematician Helmert first derived the distribution of the sample variance for normally distributed data. He showed that $(n-1)S^2/\sigma^2$ follows what we now call the chi-square distribution.

Karl Pearson (1900)

The breakthrough came when Pearson published "On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling."

Pearson's Problem: Are the deviations I see in my data just due to random chance, or is there something systematic going on?

His motivation was practical: testing whether dice were fair, validating Mendel's genetic inheritance data, and checking if theoretical distributions fit observed data.

Ronald Fisher (1920s)

Fisher rigorously developed the concept of degrees of freedom and showed how to apply chi-square tests correctly. He established that when you estimate parameters from data, you "use up" degrees of freedom.

Why Do We Need the Chi-Square Distribution?

Chi-square is the foundation of statistical testing. It answers fundamental questions about data:

Mathematical Definition

Definition from Normal Variables

If $Z_1, Z_2, \ldots, Z_k$ are independent standard normal random variables, then:

We say $X$ follows a chi-square distribution with k degrees of freedom.

The Probability Density Function

Symbol by Symbol

Connection to Gamma Distribution

The chi-square distribution is a special case of Gamma:

This means: shape parameter $\alpha = k/2$, rate parameter $\beta = 1/2$.

Exploring the Distribution

Use the interactive visualizer below to see how the degrees of freedom $k$ affects the shape of the chi-square distribution.

Key Observations

• k = 1: Heavily right-skewed, mode at 0, unbounded spike at origin
• k = 2: Exponential distribution! Mode at 0 but no spike
• k ≥ 3: Interior mode at $k - 2$, bell-like but still skewed
• Large k: Becomes more symmetric, approximates Normal by CLT
• Mean always equals k: Larger k shifts distribution right

Where Chi-Square Comes From

The chi-square distribution arises naturally when you square and sum independent standard normal variables. This interactive demonstration shows you why.

This fundamental relationship is why chi-square appears everywhere in statistics: any time you're working with squared errors or deviations from normality, chi-square is lurking.

Key Properties

Summary Statistics

Additivity Property

If $X_1 \sim \chi^2_{k_1}$ and $X_2 \sim \chi^2_{k_2}$ are independent, then:

Why this matters: You can decompose complex chi-square statistics into simpler components, or combine independent tests.

Scaling Property

If $X \sim \chi^2_k$ and $c > 0$:

Scaling doesn't preserve chi-square (unlike the Normal), but stays in the Gamma family.

The Chi-Square Test Framework

The chi-square test compares observed frequencies to expected frequencies using this statistic:

What Each Term Means

Goodness-of-Fit Test: Is This Die Fair?

The goodness-of-fit test asks: Does my observed data match a hypothesized distribution? Try it yourself:

Steps for a Goodness-of-Fit Test

1. State hypotheses: $H_0$: Data follows expected distribution
2. Calculate expected frequencies under $H_0$
3. Compute chi-square statistic: $\chi^2 = \sum (O_i - E_i)^2 / E_i$
4. Determine degrees of freedom: $k = \text{categories} - 1$ (subtract 1 more for each estimated parameter)
5. Find p-value from $\chi^2_k$ distribution
6. Decide: Reject $H_0$ if p-value < significance level

Real-World Applications

AI/ML Applications

Chi-square is a fundamental tool in machine learning, especially for categorical data analysis and feature selection.

1. Chi-Square Feature Selection

One of the most important uses of chi-square in ML is selecting the most informative features for classification. The idea: if a feature is independent of the target, it's not useful for prediction.

from sklearn.feature_selection import chi2, SelectKBest

# Select top 3 features by chi-square score
selector = SelectKBest(chi2, k=3)
X_selected = selector.fit_transform(X, y)

# Selected features: income_level, age_group, device

2. Text Classification & NLP

Chi-square is heavily used in text classification to:

• Select the most discriminative words/n-grams for each class
• Reduce vocabulary size while preserving predictive power
• Identify terms strongly associated with categories

1from sklearn.feature_extraction.text import CountVectorizer
2from sklearn.feature_selection import chi2, SelectKBest
3
4# Create bag-of-words features
5vectorizer = CountVectorizer(max_features=10000)
6X = vectorizer.fit_transform(documents)
7
8# Select top 1000 features by chi-square score
9selector = SelectKBest(chi2, k=1000)
10X_selected = selector.fit_transform(X, y)
11
12# Get the selected feature names
13feature_names = vectorizer.get_feature_names_out()
14selected_features = feature_names[selector.get_support()]

3. Categorical Encoding Decisions

When deciding how to encode categorical features, chi-square helps determine:

• Whether a categorical feature is worth including
• Which categories to merge (similar chi-square contributions)
• Target encoding quality assessment

4. Model Calibration Testing

Chi-square goodness-of-fit can test if a model's predicted probabilities match reality:

• Bin predictions into groups (e.g., 0-10%, 10-20%, ...)
• Compare predicted vs. actual positive rates in each bin
• Large chi-square indicates poor calibration

5. Independence in Graphical Models

In probabilistic graphical models and causal inference, chi-square tests help:

• Test conditional independence assumptions
• Validate Bayesian network structure
• Identify spurious correlations vs. causal relationships

Related Distributions

Python Implementation

Basic Operations with SciPy

1import numpy as np
2from scipy import stats
3
4# Create chi-square distribution with k=5 degrees of freedom
5k = 5
6chi2_dist = stats.chi2(df=k)
7
8# PDF, CDF, quantiles
9x = 5
10print(f"PDF at x={x}: {chi2_dist.pdf(x):.4f}")
11print(f"CDF at x={x}: {chi2_dist.cdf(x):.4f}")
12print(f"P(X > {x}): {1 - chi2_dist.cdf(x):.4f}")
13
14# Critical values for common significance levels
15print(f"95th percentile: {chi2_dist.ppf(0.95):.4f}")
16print(f"99th percentile: {chi2_dist.ppf(0.99):.4f}")
17
18# Summary statistics
19print(f"Mean: {chi2_dist.mean():.4f}")  # Should equal k
20print(f"Variance: {chi2_dist.var():.4f}")  # Should equal 2k
21
22# Generate random samples
23samples = chi2_dist.rvs(size=10000)
24print(f"Sample mean: {samples.mean():.4f}")
25print(f"Sample variance: {samples.var():.4f}")

Goodness-of-Fit Test

1import numpy as np
2from scipy import stats
3
4# Example: Testing if a die is fair
5# Observed counts from 600 rolls
6observed = np.array([90, 105, 115, 95, 85, 110])
7
8# Expected counts if fair (600/6 = 100 each)
9expected = np.array([100, 100, 100, 100, 100, 100])
10
11# Method 1: Using scipy.stats.chisquare
12stat, pvalue = stats.chisquare(observed, f_exp=expected)
13print(f"Chi-square statistic: {stat:.4f}")
14print(f"P-value: {pvalue:.4f}")
15print(f"Degrees of freedom: {len(observed) - 1}")
16
17# Method 2: Manual calculation
18chi_sq_manual = np.sum((observed - expected)**2 / expected)
19df = len(observed) - 1
20pvalue_manual = 1 - stats.chi2.cdf(chi_sq_manual, df)
21print(f"Manual calculation: χ² = {chi_sq_manual:.4f}, p = {pvalue_manual:.4f}")
22
23# Decision at α = 0.05
24alpha = 0.05
25critical_value = stats.chi2.ppf(1 - alpha, df)
26print(f"Critical value at α={alpha}: {critical_value:.4f}")
27print(f"Reject H₀: {stat > critical_value}")

Test of Independence

1import numpy as np
2from scipy import stats
3
4# Contingency table: Gender vs. Product Preference
5# Rows: Male, Female
6# Columns: Product A, Product B, Product C
7contingency_table = np.array([
8    [45, 35, 20],   # Male
9    [30, 50, 25]    # Female
10])
11
12# Perform chi-square test of independence
13stat, pvalue, dof, expected = stats.chi2_contingency(contingency_table)
14
15print("Observed frequencies:")
16print(contingency_table)
17print("\nExpected frequencies (under independence):")
18print(expected.round(2))
19print(f"\nChi-square statistic: {stat:.4f}")
20print(f"Degrees of freedom: {dof}")
21print(f"P-value: {pvalue:.4f}")
22
23# Calculate contributions from each cell
24contributions = (contingency_table - expected)**2 / expected
25print("\nContributions to chi-square:")
26print(contributions.round(4))

Feature Selection with Chi-Square

1from sklearn.datasets import load_iris
2from sklearn.feature_selection import chi2, SelectKBest
3import numpy as np
4
5# Load data (using discretized version for chi-square)
6iris = load_iris()
7X = iris.data
8y = iris.target
9
10# Chi-square requires non-negative features
11# For continuous features, you might bin them first
12X_positive = X - X.min(axis=0)  # Shift to non-negative
13
14# Calculate chi-square scores for each feature
15scores, pvalues = chi2(X_positive, y)
16
17print("Feature chi-square scores:")
18for i, (name, score, pval) in enumerate(zip(
19    iris.feature_names, scores, pvalues
20)):
21    print(f"  {name}: χ² = {score:.2f}, p = {pval:.4f}")
22
23# Select top k features
24k = 2
25selector = SelectKBest(chi2, k=k)
26X_selected = selector.fit_transform(X_positive, y)
27
28# Which features were selected?
29selected_mask = selector.get_support()
30selected_features = np.array(iris.feature_names)[selected_mask]
31print(f"\nSelected features: {selected_features}")

Common Pitfalls

Test Your Understanding

Summary

The chi-square distribution is foundational to statistical testing. It naturally arises from squared standard normal variables and provides the basis for testing goodness-of-fit, independence, and variance.

Key Formulas

Key Takeaways

1. Sum of squared normals: Chi-square arises naturally from squaring and summing independent standard normal variables
2. Degrees of freedom = k: This single parameter controls shape, mean, and variance
3. Special case of Gamma: $\chi^2_k = \text{Gamma}(k/2, 1/2)$
4. Goodness-of-fit: Test if observed data matches expected distribution
5. Independence test: Test if categorical variables are related
6. ML feature selection: Chi-square score ranks categorical features by predictive power
7. Foundation for other tests: t-test, F-test, ANOVA all use chi-square

Coming Next: In the next section, we'll explore the Student's t-distribution—essential for inference about means when the population variance is unknown.