AI Book - Master Artificial Intelligence by Building from Scratch

Learning Objectives

By the end of this section, you will:

Understand the moments framework and why higher moments matter for describing distribution shapes
Master skewness (3rd moment): measure asymmetry, interpret positive/negative values, and recognize real-world examples
Master kurtosis (4th moment): understand tail behavior, distinguish leptokurtic/mesokurtic/platykurtic distributions
Debunk the common misconception that kurtosis measures "peakedness"
Apply these concepts to AI/ML problems: feature engineering, data preprocessing, anomaly detection, and model diagnostics
Calculate and interpret skewness and kurtosis in Python
Understand the mathematical relationship between skewness and kurtosis

Historical Context

Beyond Mean and Variance: The Quest for Shape

By the late 19th century, statisticians realized that two numbers—mean and variance—weren't enough to fully describe a distribution. Different distributions could have identical means and variances yet look completely different!

Karl Pearson (1857-1936) pioneered the systematic study of distribution shapes. He coined the terms "skewness" and "kurtosis" while developing his famous Pearson system of distributions, which classified distributions by their higher moments.

Pearson's work was driven by practical needs: biological measurements often showed asymmetric patterns that the normal distribution couldn't capture. He needed mathematical tools to describe these departures from normality.

📐

1st Moment

Mean (location)

↔️

2nd Moment

Variance (spread)

🔄

3rd & 4th Moments

Shape (skewness, kurtosis)

The Problem: Shape Matters

Consider these three distributions. They all have the same mean (μ = 0) and the same variance (σ² = 1). Yet they represent completely different data patterns:

Left-Skewed

⬅️ 📊

Long tail on the left. Example: Student test scores on an easy exam (most score high, few score very low).

Symmetric

📊

Equal tails on both sides. Example: Measurement errors in physics experiments.

Right-Skewed

📊 ➡️

Long tail on the right. Example: Income distribution (most earn average, few earn extremely high).

The Core Insight: Mean tells us WHERE the distribution is centered. Variance tells us HOW SPREAD OUT it is. But we need more information to know the SHAPE: Is it symmetric? Does it have heavy tails? That's where skewness and kurtosis come in.

The Moments Framework

Before diving into skewness and kurtosis, let's understand the unified framework: moments. Moments are summary statistics that capture different aspects of a distribution.

Types of Moments

Raw Moments (about the origin)

\mu'_r = E[X^r]

The r-th power of X, averaged. The 1st raw moment is the mean.

Central Moments (about the mean)

\mu_r = E[(X - \mu)^r]

Deviations from the mean, raised to power r. The 2nd central moment is variance.

Standardized Moments (unitless)

\tilde{\mu}_r = E\left[\left(\frac{X - \mu}{\sigma}\right)^r\right] = \frac{\mu_r}{\sigma^r}

Standardized by dividing by σ^r. This makes the moment unitless and comparable across distributions.

Moment	Formula	Measures	Name
1st raw	E[X]	Location (center)	Mean
2nd central	E[(X-μ)²]	Spread (dispersion)	Variance
3rd standardized	E[(X-μ)³]/σ³	Asymmetry	Skewness
4th standardized	E[(X-μ)⁴]/σ⁴	Tail weight	Kurtosis

Pattern: Each moment captures a different "dimension" of the distribution. You need all four to have a complete picture of location, spread, asymmetry, and tail behavior.

Interactive: Moments Calculator

Drag data points to see how each moment changes. Pay attention to how outliers affect higher moments more dramatically!

Loading interactive demo...

Try this: Add an extreme outlier and watch how kurtosis changes much more than skewness, which changes more than variance. Higher moments are progressively more sensitive to extreme values!

Skewness: Measuring Asymmetry

Definition: Skewness (3rd Standardized Moment)

\gamma_1 = E\left[\left(\frac{X - \mu}{\sigma}\right)^3\right] = \frac{E[(X-\mu)^3]}{\sigma^3} = \frac{\mu_3}{\sigma^3}

Skewness measures the asymmetry of a distribution

Why the Third Power?

The third power is odd, which means it preserves the sign of deviations:

Positive deviations (above mean) contribute positive values
Negative deviations (below mean) contribute negative values

If the right tail is longer, there are more extreme positive deviations, so skewness is positive. If the left tail is longer, skewness is negative.

Interpreting Skewness

γ₁ < 0

⬅️📊

Left-skewed (negative skew). Long left tail. Mean < Median < Mode.

γ₁ ≈ 0

📊

Symmetric. Equal tails. Mean ≈ Median ≈ Mode.

γ₁ > 0

📊➡️

Right-skewed (positive skew). Long right tail. Mode < Median < Mean.

Rule of Thumb: |γ₁| < 0.5 is approximately symmetric, 0.5 ≤ |γ₁| < 1 is moderately skewed, |γ₁| ≥ 1 is highly skewed.

Interactive: Skewness Visualizer

Adjust the skewness parameter to see how it affects the distribution shape. Watch how mean, median, and mode separate for skewed distributions!

Loading interactive demo...

Real-World Examples of Skewness

Right-Skewed (Positive Skewness)

💰 Income Distribution

Most people earn moderate incomes, but a few billionaires create a long right tail. Median income is lower than mean income.

⏱️ Response Times

Most API calls complete quickly, but some take much longer due to network issues or server load.

🏠 House Prices

Most houses cost moderate amounts, but luxury properties can cost orders of magnitude more.

📊 Word Frequencies

In any text, a few words appear very frequently while most words are rare (Zipf's law).

Left-Skewed (Negative Skewness)

📝 Easy Exam Scores

Most students score high (near 100%), but a few score much lower, creating a left tail.

🎂 Age at Death (Developed Countries)

Most people live to old age, but some die young, creating a left tail.

Finance: A Special Case

Stock Returns: Daily stock returns typically show negative skewness. This means large losses are more common than equally large gains. A "crash" of -10% happens more often than a "rally" of +10%. This asymmetric risk is why risk management in finance is so critical!

Kurtosis: Measuring Tail Behavior

Definition: Kurtosis (4th Standardized Moment)

\gamma_2 = E\left[\left(\frac{X - \mu}{\sigma}\right)^4\right] = \frac{E[(X-\mu)^4]}{\sigma^4} = \frac{\mu_4}{\sigma^4}

The normal distribution has kurtosis = 3

Excess Kurtosis

Since the normal distribution has kurtosis = 3, we often use excess kurtosis to compare with normal:

\text{Excess Kurtosis} = \gamma_2 - 3

Python & R convention: Most statistical software reports excess kurtosis by default. In scipy.stats.kurtosis() and R's moments::kurtosis(), excess kurtosis is the default.

Classification by Kurtosis

Platykurtic (κ < 0)

🔻

Lighter tails than normal. Fewer extreme values. Example: Uniform distribution.

"Platy" = flat/broad (Greek)

Mesokurtic (κ ≈ 0)

📊

Normal-like tails. The reference point. Example: Normal distribution.

"Meso" = middle (Greek)

Leptokurtic (κ > 0)

🔺

Heavier tails than normal. More extreme values. Example: t-distribution, Laplace.

"Lepto" = slender (Greek)

Interactive: Kurtosis Visualizer

Compare distributions with different kurtosis values. Pay special attention to the tail probabilities—this is where kurtosis really matters!

Loading interactive demo...

The Kurtosis Misconception

COMMON MISCONCEPTION: Kurtosis measures "peakedness".

THE TRUTH: Kurtosis primarily measures tail weight, not the height of the peak! This is one of the most widespread misunderstandings in statistics.

The confusion arose because textbooks often showed distributions getting "sharper" as kurtosis increased. But this is misleading. Here's why:

The 4th power heavily weights tail values: Raising to the 4th power amplifies extreme values far more than values near the center.
Counterexamples exist: You can construct distributions with high kurtosis that are NOT particularly peaked, and vice versa.
Practical implications: Kurtosis tells you about the probability of extreme events (outliers), not about the shape near the center.

For AI/ML Engineers: When you see high kurtosis in your data, think "potential outliers" and "extreme events more likely than normal"—not "peaked distribution". This affects how you handle preprocessing, choose loss functions, and interpret model predictions.

Interactive: Distribution Shape Space

Explore the relationship between skewness and kurtosis across different distributions. Notice how they cluster in specific regions of this space!

Loading interactive demo...

Mathematical Constraint: Not all (skewness, kurtosis) pairs are possible. The relationship κ ≥ γ₁² - 2 (approximately) must hold. This is why there's an "impossible region" in the plot.

Common Distributions Reference

Distribution	Skewness (γ₁)	Excess Kurtosis (κ)	Notes
Normal	0	0	The reference (mesokurtic)
Uniform	0	-1.2	Lightest possible tails (platykurtic)
Exponential(λ)	2	6	Always right-skewed, heavy tails
Laplace	0	3	Symmetric but heavy tails
Bernoulli(0.5)	0	-2	Symmetric, very light tails
Poisson(λ)	1/√λ	1/λ	Skewness decreases with λ
Student t(ν)	0 (ν>3)	6/(ν-4) (ν>4)	Heavy tails, undefined for small ν
Log-Normal(μ,σ)	depends on σ	depends on σ	Always right-skewed
Chi-squared(k)	√(8/k)	12/k	Right-skewed, approaches normal

Poisson insight: As λ increases, Poisson becomes more symmetric and more normal-like (both skewness and kurtosis approach 0).

AI/ML Applications

1. Feature Engineering

Skewness and kurtosis themselves can be useful features for tabular ML models:

🐍python

1# Feature engineering: add skewness/kurtosis of rolling window
2df['price_skew_7d'] = df['price'].rolling(7).apply(lambda x: skew(x))
3df['price_kurt_7d'] = df['price'].rolling(7).apply(lambda x: kurtosis(x))

High kurtosis in a rolling window might indicate upcoming volatility. Changing skewness might signal regime changes.

2. Data Preprocessing

Highly skewed features can cause problems for many ML algorithms:

Gradient-based methods may converge slowly
Distance-based algorithms (k-NN, SVM) become dominated by skewed features
Linear models may fit poorly

Solution: Transform skewed features using log, sqrt, or Box-Cox:

🐍python

1from scipy.stats import skew, boxcox
2
3# Check skewness
4print(f"Original skewness: {skew(data):.3f}")
5
6# Apply log transform if right-skewed
7if skew(data) > 1:
8    data_transformed = np.log1p(data)  # log(1 + x) handles zeros
9
10# Or use Box-Cox for optimal transformation
11data_transformed, lambda_param = boxcox(data + 1)
12print(f"Transformed skewness: {skew(data_transformed):.3f}")

3. Anomaly Detection

High kurtosis = more outliers! If your data has high kurtosis and you use Z-scores for anomaly detection, you'll flag more points than expected:

🐍python

1from scipy.stats import kurtosis
2
3# Check kurtosis
4k = kurtosis(data)  # excess kurtosis
5print(f"Excess kurtosis: {k:.3f}")
6
7if k > 1:
8    print("Warning: Heavy tails detected!")
9    print("Consider robust methods (MAD, IQR) instead of Z-scores")
10
11    # Robust anomaly detection
12    from scipy.stats import median_abs_deviation
13    mad = median_abs_deviation(data)
14    threshold = 3 * mad
15    anomalies = np.abs(data - np.median(data)) > threshold

4. Neural Network Training

Gradient distributions during training should ideally be symmetric and not too heavy-tailed:

High kurtosis gradients → potential exploding gradient problem
Highly skewed gradients → uneven parameter updates

Monitor gradient statistics during training for diagnostic purposes!

5. Model Diagnostics

Residuals from regression should ideally be symmetric (skewness ≈ 0) and have normal-like tails (kurtosis ≈ 0):

🐍python

1# Check residual distribution
2from scipy.stats import skew, kurtosis, normaltest
3
4residuals = y_true - y_pred
5print(f"Residual skewness: {skew(residuals):.3f}")
6print(f"Residual kurtosis: {kurtosis(residuals):.3f}")
7
8# Formal test for normality
9stat, p_value = normaltest(residuals)
10if p_value < 0.05:
11    print("Warning: Residuals significantly non-normal")

Python Implementation

🐍python

1import numpy as np
2from scipy import stats
3
4# ============================================
5# CALCULATING SKEWNESS AND KURTOSIS
6# ============================================
7
8# Sample data
9data = np.array([2, 3, 5, 6, 6, 6, 7, 8, 9, 20])  # Right-skewed with outlier
10
11# --- From the Definition ---
12def skewness_manual(x):
13    """Calculate skewness from definition."""
14    n = len(x)
15    mu = np.mean(x)
16    sigma = np.std(x, ddof=0)  # Population std
17    return np.mean(((x - mu) / sigma) ** 3)
18
19def kurtosis_manual(x, excess=True):
20    """Calculate (excess) kurtosis from definition."""
21    n = len(x)
22    mu = np.mean(x)
23    sigma = np.std(x, ddof=0)
24    k = np.mean(((x - mu) / sigma) ** 4)
25    return k - 3 if excess else k
26
27# --- Using SciPy ---
28from scipy.stats import skew, kurtosis
29
30# Population formulas (bias=False is default, uses N-1)
31skew_scipy = skew(data)
32kurt_scipy = kurtosis(data)  # Excess kurtosis by default!
33
34print(f"Skewness: {skew_scipy:.4f}")
35print(f"Excess Kurtosis: {kurt_scipy:.4f}")
36
37# ============================================
38# CHECKING DISTRIBUTIONS
39# ============================================
40
41# Generate samples from different distributions
42np.random.seed(42)
43n = 10000
44
45# Normal (symmetric, mesokurtic)
46normal_data = np.random.normal(0, 1, n)
47print(f"\nNormal: skew={skew(normal_data):.3f}, kurt={kurtosis(normal_data):.3f}")
48
49# Exponential (right-skewed, leptokurtic)
50exp_data = np.random.exponential(1, n)
51print(f"Exponential: skew={skew(exp_data):.3f}, kurt={kurtosis(exp_data):.3f}")
52
53# Uniform (symmetric, platykurtic)
54uniform_data = np.random.uniform(-1, 1, n)
55print(f"Uniform: skew={skew(uniform_data):.3f}, kurt={kurtosis(uniform_data):.3f}")
56
57# t-distribution (symmetric, heavy tails)
58t_data = np.random.standard_t(df=5, size=n)
59print(f"t(df=5): skew={skew(t_data):.3f}, kurt={kurtosis(t_data):.3f}")
60
61# ============================================
62# TRANSFORMATION TO REDUCE SKEWNESS
63# ============================================
64
65from scipy.stats import boxcox
66
67# Right-skewed data (e.g., income)
68income_data = np.random.exponential(50000, 1000)
69print(f"\nOriginal income skewness: {skew(income_data):.3f}")
70
71# Log transform (simple)
72log_income = np.log1p(income_data)
73print(f"Log-transformed skewness: {skew(log_income):.3f}")
74
75# Box-Cox transform (optimal)
76income_positive = income_data + 1  # Must be positive
77bc_income, lambda_param = boxcox(income_positive)
78print(f"Box-Cox skewness: {skew(bc_income):.3f} (λ={lambda_param:.3f})")
79
80# ============================================
81# SAMPLE SIZE CONSIDERATIONS
82# ============================================
83
84# Higher moments need more data!
85sample_sizes = [20, 50, 100, 500, 1000, 10000]
86
87for n in sample_sizes:
88    samples = np.random.exponential(1, n)
89    # True skewness of exponential is 2, true excess kurtosis is 6
90    print(f"n={n:5d}: skew={skew(samples):6.3f}, kurt={kurtosis(samples):6.3f}")

Sample Size Rule of Thumb: Skewness estimates become reasonably stable around n=50. Kurtosis estimates need n>100, and ideally n>500 for reliability. With small samples, treat higher moment estimates with caution!

Common Pitfalls

Pitfall 1: Confusing Kurtosis with "Peakedness"

As discussed above, kurtosis measures tail weight, NOT how peaked the distribution is. A distribution can be flat-topped yet have high kurtosis due to heavy tails.

Pitfall 2: Using Excess vs Raw Kurtosis Inconsistently

SciPy reports excess kurtosis (κ - 3) by default. Excel's KURT() function also returns excess kurtosis. But some sources report raw kurtosis. Always check documentation!

Pitfall 3: Insufficient Sample Size

Higher moments are increasingly sensitive to sample size. Kurtosis estimates with n < 50 are often unreliable. Use bootstrapping to assess uncertainty in your estimates.

Pitfall 4: Outlier Hypersensitivity

A single extreme outlier can massively inflate kurtosis. If you suspect outliers, use robust measures like the medcouple for skewness or compare trimmed vs untrimmed estimates.

Pitfall 5: Assuming Symmetry Implies Zero Skewness

For populations, symmetry implies zero skewness. But sample skewness from a symmetric population won't be exactly zero—you need hypothesis testing to conclude anything about population skewness.

Pitfall 6: Ignoring Undefined Moments

Some distributions have undefined higher moments! The Cauchy distribution has undefined mean, variance, skewness, and kurtosis. The t-distribution with df=3 has undefined kurtosis. Always verify moments exist for your distribution.

Test Your Understanding

Take this quiz to check your understanding of skewness and kurtosis!

Loading interactive demo...

Summary

Key Takeaways

Moments framework: Mean (1st), Variance (2nd), Skewness (3rd), Kurtosis (4th) each capture a different aspect of a distribution.
Skewness (γ₁) measures asymmetry: positive = right tail longer, negative = left tail longer, zero = symmetric.
Kurtosis (γ₂) measures tail heaviness, NOT peakedness! Leptokurtic (heavy tails), mesokurtic (normal-like), platykurtic (light tails).
Excess kurtosis = kurtosis - 3 is used because normal distribution has kurtosis = 3.
Higher moments are more sensitive to outliers and require larger samples for reliable estimation.
AI/ML applications: feature engineering, preprocessing (transforming skewed features), anomaly detection, model diagnostics.
Stock returns typically show negative skewness (crashes) and positive excess kurtosis (fat tails)—important for risk management.
Not all (skewness, kurtosis) combinations are mathematically possible: κ ≥ γ₁² - 2 approximately.

Final Thought: Mean and variance tell you WHERE and HOW SPREAD a distribution is. Skewness and kurtosis tell you its SHAPE. Together, these four moments give you a complete statistical fingerprint of any distribution— essential knowledge for any AI/ML engineer working with real-world data.

Next Up: In the next section, we'll learn about Moment Generating Functions (MGFs)—a powerful mathematical tool that encodes ALL moments in a single function and connects to the Central Limit Theorem!