Shannon Entropy

Learning Objectives

By the end of this section, you will be able to:

Where You'll Apply This: Data compression algorithms, decision tree learning, neural network loss functions, text generation (temperature sampling), anomaly detection, feature selection, and understanding model uncertainty.

The Big Picture

How do you measure information? This seemingly philosophical question has a precise mathematical answer that revolutionized communication, computing, and now machine learning. Shannon entropy is the fundamental measure of uncertainty, information content, and surprise in a probability distribution.

Historical Context

The story of entropy begins with one of the most influential scientific papers ever written.

Building Block: Surprisal (Self-Information)

Before defining entropy, we need to understand its building block: surprisal(also called self-information). Surprisal measures how "surprising" a single event is.

Why this formula? Shannon derived it from three intuitive requirements:

1. Rare events are more surprising: An event with low probability should have high surprisal. If $p(x) \to 0$, then $I(x) \to \infty$.
2. Certain events carry no information: If $p(x) = 1$, then $I(x) = 0$. Learning something you already knew is not informative.
3. Independent events add: The surprisal of two independent events occurring should be the sum of their individual surprisals. This requires a logarithm!$I(x, y) = I(x) + I(y)$ when x and y are independent.

Interactive: Surprisal Explorer

Explore how surprisal changes with probability. Notice how rare events carry much more information than common events.

Shannon Entropy Definition

Shannon entropy is the expected surprisal - the average amount of information gained when observing a random variable. It measures the uncertainty inherent in a probability distribution.

Binary Entropy

The simplest case is a binary random variable (like a coin flip). The binary entropy function H(p) gives the entropy as a function of the probability p of one outcome:

Interactive: Binary Entropy

Drag the slider to see how entropy changes with bias. Notice that entropy is maximized at p = 0.5 (fair coin) and approaches zero as the coin becomes more biased.

Discrete Entropy

For a discrete distribution over n outcomes, entropy measures total uncertainty. You can manipulate the probabilities directly and see how entropy responds.

Interactive: Discrete Entropy Explorer

Properties of Entropy

Shannon entropy has several important mathematical properties that make it the unique measure of information:

Maximum Entropy Principle

A fundamental result: among all distributions over n outcomes, the uniform distribution maximizes entropy. This is because uniform represents maximum uncertainty - we have no reason to prefer any outcome over another.

Interactive: Maximum Entropy Demonstration

Watch entropy increase as a skewed distribution transforms into a uniform distribution. This demonstrates why uniform = maximum entropy.

Entropy Comparison Chart

To build intuition, let's compare entropy values across different common distributions. This helps you develop a sense for what different entropy values "feel like."

Real-World Examples

AI/ML Connections

Shannon entropy is everywhere in modern machine learning. Understanding it deeply will make you a better ML practitioner.

Python Implementation

Let's implement entropy calculation in Python. This code demonstrates both the basic formula and practical considerations.

For p = [0.5, 0.5]: -0.5*log₂(0.5) - 0.5*log₂(0.5) = 1 bit

1import numpy as np
2
3# Basic entropy calculation
4def shannon_entropy(probabilities):
5    """
6    Calculate Shannon entropy of a discrete distribution.
7
8    Parameters:
9        probabilities: Array-like of probabilities (must sum to 1)
10
11    Returns:
12        Entropy in bits (log base 2)
13    """
14    # Convert to numpy array
15    p = np.array(probabilities, dtype=np.float64)
16
17    # Normalize to ensure sum = 1
18    p = p / p.sum()
19
20    # Filter out zero probabilities (0 * log(0) = 0 by convention)
21    p_nonzero = p[p > 0]
22
23    # Calculate entropy: H = -sum(p * log2(p))
24    return -np.sum(p_nonzero * np.log2(p_nonzero))

1import numpy as np
2from scipy.stats import entropy as scipy_entropy
3
4def shannon_entropy(probabilities):
5    """Calculate Shannon entropy in bits."""
6    p = np.array(probabilities, dtype=np.float64)
7    p = p / p.sum()
8    p_nonzero = p[p > 0]
9    return -np.sum(p_nonzero * np.log2(p_nonzero))
10
11# ============================================
12# Example 1: Binary distributions
13# ============================================
14print("=== Binary Entropy ===")
15for p in [0.5, 0.7, 0.9, 0.99]:
16    h = shannon_entropy([p, 1-p])
17    print(f"p={p:.2f}: H = {h:.4f} bits")
18
19# Output:
20# p=0.50: H = 1.0000 bits
21# p=0.70: H = 0.8813 bits
22# p=0.90: H = 0.4690 bits
23# p=0.99: H = 0.0808 bits
24
25# ============================================
26# Example 2: Compare with SciPy
27# ============================================
28print("\n=== SciPy Comparison ===")
29fair_die = [1/6] * 6
30print(f"Our implementation: {shannon_entropy(fair_die):.4f} bits")
31print(f"SciPy (base 2):     {scipy_entropy(fair_die, base=2):.4f} bits")
32
33# ============================================
34# Example 3: Information gain for decision trees
35# ============================================
36def information_gain(parent_labels, child_groups):
37    """
38    Calculate information gain for a split.
39
40    Parameters:
41        parent_labels: Array of class labels before split
42        child_groups: List of arrays of class labels for each child
43    """
44    # Parent entropy
45    _, counts = np.unique(parent_labels, return_counts=True)
46    h_parent = shannon_entropy(counts / len(parent_labels))
47
48    # Weighted child entropy
49    n_total = len(parent_labels)
50    h_children = 0
51    for child in child_groups:
52        if len(child) > 0:
53            _, counts = np.unique(child, return_counts=True)
54            h_child = shannon_entropy(counts / len(child))
55            weight = len(child) / n_total
56            h_children += weight * h_child
57
58    return h_parent - h_children
59
60# Example: Splitting data by a feature
61labels = ['A', 'A', 'A', 'A', 'B', 'B', 'B', 'B']  # 50/50 split
62left = ['A', 'A', 'A', 'B']  # 75/25
63right = ['A', 'B', 'B', 'B']  # 25/75
64
65ig = information_gain(labels, [left, right])
66print(f"\n=== Information Gain Example ===")
67print(f"Parent entropy: {shannon_entropy([0.5, 0.5]):.4f} bits")
68print(f"After split:    {ig:.4f} bits of information gained")
69
70# ============================================
71# Example 4: Text entropy estimation
72# ============================================
73def text_entropy(text):
74    """Estimate entropy of text based on character frequencies."""
75    text = text.lower()
76    chars = [c for c in text if c.isalpha()]
77    _, counts = np.unique(chars, return_counts=True)
78    return shannon_entropy(counts / sum(counts))
79
80sample_text = "the quick brown fox jumps over the lazy dog"
81h = text_entropy(sample_text)
82max_h = np.log2(26)  # Uniform over 26 letters
83print(f"\n=== Text Entropy ===")
84print(f"Sample text entropy: {h:.4f} bits/char")
85print(f"Max possible:        {max_h:.4f} bits/char")
86print(f"Compression ratio:   {h/max_h:.1%}")

Knowledge Check

Test your understanding of Shannon entropy with this interactive quiz.

Summary

Looking Ahead: In the next section, we'll explore cross-entropy, which measures the "distance" between two distributions. This is the foundation of the classification loss functions used in every neural network!