AI Book - Master Artificial Intelligence by Building from Scratch

Learning Objectives

By the end of this section, you will:

Understand what a random variable truly is: a function from sample space to real numbers
Distinguish between the sample space (outcomes) and the range (numerical values)
Master the formal definition $X: \Omega o \mathbb{R}$ and its intuitive meaning
Identify discrete random variables by their countable range
Apply random variables to model real-world experiments
Connect to AI/ML: classification outputs, token predictions, discrete actions in RL

Historical Context

The Birth of Random Variables: From Games to Mathematics

In the early days of probability (17th-18th century), mathematicians like Blaise Pascal, Pierre de Fermat, and Jacob Bernoulli worked directly with sample spaces and events. But they faced a fundamental problem...

The Problem: How do you do arithmetic with random outcomes? You can't add "Heads" + "Tails"!

The Solution: Abraham de Moivre (1667-1754) began assigning numbers to outcomes, and Andrey Kolmogorov (1903-1987) formalized this in his 1933 axioms as the concept of a random variable—a function that translates outcomes into numbers we can compute with.

🎲

Outcome

"Die shows 5"

→

5️⃣

Number

X(ω) = 5

Why Do We Need Random Variables?

Imagine you flip a coin. The sample space is $\Omega = \{ ext{Heads}, ext{Tails}\}$ . Now try to answer these questions:

What is the "average" outcome?
What is the variance of the outcomes?
Can you graph a histogram of outcomes?

You can't! "Heads" and "Tails" are labels, not numbers. We need a way to convert outcomes into numbers so we can:

Calculate expected values (averages)
Measure variance and standard deviation
Build probability distributions
Use calculus and linear algebra in probability
Train machine learning models on probabilistic outputs

The Key Insight: A random variable is a "translator" that converts outcomes (which may be anything—coin faces, card suits, weather conditions) into real numbers that we can compute with.

Formal Definition: X: Ω → ℝ

Definition: Random Variable

X: \Omega o \mathbb{R}

A random variable X is a function that assigns a real number to each outcome in the sample space.

Breaking Down the Notation

Symbol	Name	Meaning
X	Random Variable	The function that does the mapping
Ω (Omega)	Sample Space	Set of all possible outcomes of the experiment
ℝ	Real Numbers	The target set—all possible numerical values
ω (omega)	Outcome	A single element of the sample space
X(ω)	Value of X at ω	The number assigned to outcome ω

Two Metaphors to Understand Random Variables

🔄 The Translator Metaphor

A random variable translates the "language" of outcomes into the "language" of numbers. Input: "Heads". Output: 1.

📏 The Measurement Device Metaphor

A random variable is like a measurement tool that measures a numerical property of the outcome. Draw a card → measure its face value.

Critical Point: The random variable X is a deterministic function! The "randomness" doesn't come from X—it comes from not knowing which outcome ω will occur. Once ω is known, X(ω) is uniquely determined.

Interactive: Random Variable Mapping

Explore how a random variable maps outcomes from the sample space Ω to numerical values in ℝ. Try different experiments to see how the mapping changes!

Loading interactive demo...

What Makes a Random Variable Discrete?

Definition: Discrete Random Variable

A random variable X is discrete if its range(the set of possible values it can take) is countable.

What Does "Countable" Mean?

A set is countable if you can list its elements in a sequence (even if the sequence is infinite). This includes:

Finite sets: $\{0, 1\}$ , $\{1, 2, 3, 4, 5, 6\}$
Countably infinite sets: $\{0, 1, 2, 3, \ldots\}$ , $\{\ldots, -2, -1, 0, 1, 2, \ldots\}$

Examples of Discrete Random Variables

Random Variable	Range	Why Discrete?
Number of heads in 10 flips	{0, 1, 2, ..., 10}	Finite set (11 values)
Number of customers per hour	{0, 1, 2, 3, ...}	Countably infinite (you can list them)
Die roll	{1, 2, 3, 4, 5, 6}	Finite set (6 values)
Token ID in GPT vocabulary	{0, 1, ..., 50256}	Finite set (vocabulary size)

Discrete vs. Continuous: A First Look

In contrast, a continuous random variable can take any value in an interval. For example, the exact height of a person can be 170.523847... cm—there areuncountably many possible values. We'll cover continuous random variables in Section 3.

Interactive: Experiment Simulator

Run experiments and watch how outcomes map to random variable values. See how the empirical distribution converges to the theoretical one as you run more trials!

Loading interactive demo...

Classic Examples

Example 1: Coin Flip (Bernoulli Trial)

Experiment: Flip a fair coin

Sample Space: $\Omega = \{ ext{Heads}, ext{Tails}\}$

Random Variable X: "1 if Heads, 0 if Tails"

X( ext{Heads}) = 1, \quad X( ext{Tails}) = 0

Range: $\{0, 1\}$ — finite, so X is discrete!

This is called a Bernoulli random variable and is the building block for many distributions. In ML, binary classification outputs are Bernoulli!

Example 2: Single Die Roll

Experiment: Roll a fair 6-sided die

Sample Space: $\Omega = \{ ext{⚀}, ext{⚁}, ext{⚂}, ext{⚃}, ext{⚄}, ext{⚅}\}$

Random Variable X: "Number of dots showing"

Range: $\{1, 2, 3, 4, 5, 6\}$

Notice: The sample space could be faces of a die (physical objects), but X maps them to numbers!

Example 3: Counting Events

Experiment: Count emails received in one hour

Sample Space: Ω = all possible sequences of email arrivals

Random Variable N: "Number of emails received"

Range: $\{0, 1, 2, 3, \ldots\}$ — countably infinite!

This is still discrete because we can list 0, 1, 2, 3, ... even though the list never ends.

Interactive: Two Dice Sum Explorer

A classic example: when rolling two dice, the random variable S = d₁ + d₂ maps 36 outcomes to just 11 values (2 through 12). Explore the relationship between outcomes and sums!

Loading interactive demo...

Real-World Applications

📈 Finance

X = Number of trades per minute

Range: {0, 1, 2, ...}. Used for risk modeling, market microstructure analysis. Often modeled with Poisson distribution.

🏥 Healthcare

X = Number of patients in ER

Range: {0, 1, 2, ...}. Critical for staffing decisions, capacity planning. Queueing theory applications.

🏭 Quality Control

X = Number of defects per batch

Range: {0, 1, 2, ..., n}. Binomial or Poisson models. Six Sigma methodology relies heavily on this.

📱 Telecommunications

X = Number of dropped calls

Range: {0, 1, 2, ...}. Network quality optimization, SLA monitoring, capacity planning.

AI/ML Applications

Discrete random variables are everywhere in machine learning. Here's where you'll encounter them:

1. Classification Output

In a K-class classifier (e.g., ImageNet with 1000 classes):

Input: Image x
Output: Predicted class Y ∈ {0, 1, 2, ..., K-1}

Y is a discrete random variable! The softmax output gives the probability distribution P(Y = k | x) for each class k.

2. Language Models (Token Prediction)

In GPT-style autoregressive models:

Vocabulary V with |V| = 50,257 tokens (GPT-2)
X = next token ID ∈ {0, 1, 2, ..., 50256}

The model outputs P(X = token_id | context) — a probability distribution over a discrete random variable! Temperature sampling, top-k, and nucleus sampling all work with this discrete distribution.

3. Reinforcement Learning (Discrete Actions)

In games like Atari or Chess:

Action space A = {up, down, left, right, fire, ...}
A is a discrete random variable with policy π(a|s)

The policy network outputs $\pi(a|s) = P(A = a | ext{state} = s)$ — again, a distribution over a discrete random variable!

4. Attention Mechanisms

In hard attention (e.g., REINFORCE-style):

"Which token to attend to?"
A ∈ {1, 2, ..., sequence_length}

Soft attention uses weighted averages, but hard attention samples from a discrete distribution over positions.

Why This Matters: Understanding discrete random variables is essential for understanding loss functions (cross-entropy is defined over discrete distributions), sampling strategies (temperature, top-k, nucleus), and training objectives in modern AI systems.

Python Implementation

🐍python

1import numpy as np
2from collections import Counter
3
4# ==============================================
5# EXAMPLE 1: Define a random variable as a function
6# ==============================================
7
8def coin_flip_rv(outcome):
9    """
10    Random variable for coin flip
11    X = 1 if heads, 0 if tails
12    """
13    return 1 if outcome == 'H' else 0
14
15def dice_sum_rv(outcome):
16    """
17    Random variable for sum of two dice
18    S(d1, d2) = d1 + d2
19    """
20    d1, d2 = outcome
21    return d1 + d2
22
23# ==============================================
24# EXAMPLE 2: Find the range of a random variable
25# ==============================================
26
27# Sample space for two dice: all (d1, d2) pairs
28sample_space_dice = [
29    (d1, d2)
30    for d1 in range(1, 7)
31    for d2 in range(1, 7)
32]
33print(f"Sample space size: {len(sample_space_dice)}")  # 36
34
35# Apply RV to get all possible values
36rv_values = [dice_sum_rv(omega) for omega in sample_space_dice]
37
38# The range is the set of unique values
39rv_range = sorted(set(rv_values))
40print(f"Range of S: {rv_range}")
41# Output: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
42
43# ==============================================
44# EXAMPLE 3: Count outcomes for each value
45# ==============================================
46
47value_counts = Counter(rv_values)
48print("How many outcomes map to each sum:")
49for val in sorted(value_counts.keys()):
50    count = value_counts[val]
51    prob = count / 36
52    print(f"  S = {val}: {count} outcomes, P(S={val}) = {prob:.4f}")
53
54# ==============================================
55# EXAMPLE 4: Simulate a random variable
56# ==============================================
57
58def simulate_rv(rv_function, sample_space, n_samples=10000):
59    """
60    Simulate a random variable by:
61    1. Randomly selecting outcomes from sample space
62    2. Applying the RV function to each outcome
63    """
64    # Uniformly sample from sample space
65    indices = np.random.randint(0, len(sample_space), n_samples)
66    outcomes = [sample_space[i] for i in indices]
67
68    # Apply RV to get values
69    return [rv_function(omega) for omega in outcomes]
70
71# Simulate 10000 dice sums
72simulated_values = simulate_rv(dice_sum_rv, sample_space_dice, 10000)
73
74print(f"\nSimulation results (n=10000):")
75print(f"  Sample mean: {np.mean(simulated_values):.3f}")
76print(f"  Theoretical mean: 7.000")
77print(f"  Sample std: {np.std(simulated_values):.3f}")
78
79# ==============================================
80# EXAMPLE 5: ML application - token prediction
81# ==============================================
82
83def sample_from_softmax(logits, temperature=1.0):
84    """
85    Sample a discrete random variable (token ID) from logits
86    This is exactly what happens in language model generation!
87    """
88    # Apply temperature scaling
89    scaled_logits = logits / temperature
90
91    # Convert to probabilities (softmax)
92    exp_logits = np.exp(scaled_logits - np.max(scaled_logits))
93    probs = exp_logits / np.sum(exp_logits)
94
95    # Sample from discrete distribution
96    # X is a discrete RV with range {0, 1, ..., vocab_size-1}
97    token_id = np.random.choice(len(probs), p=probs)
98
99    return token_id, probs[token_id]
100
101# Example: vocabulary of 5 tokens
102logits = np.array([2.0, 1.0, 0.5, -1.0, 0.0])
103token, prob = sample_from_softmax(logits, temperature=1.0)
104print(f"\nToken prediction example:")
105print(f"  Sampled token ID: {token}")
106print(f"  Probability: {prob:.4f}")

Common Pitfalls

Pitfall 1: Confusing Outcome with Value

The outcome ω is what happens (e.g., "roll a 5"). The valueX(ω) is the number assigned (e.g., 5). These are conceptually different! The outcome lives in Ω; the value lives in ℝ.

Pitfall 2: Thinking X is Random

The function X is completely deterministic. X("Heads") is always 1. The "randomness" comes from not knowing which ω will occur, not from X itself!

Pitfall 3: Discrete Means Only Integers

Discrete means countable, not necessarily integers! A random variable taking values {0.5, 1.5, 2.5} is discrete. A random variable taking all values in [0, 1] is continuous, even though some values are "nice" numbers.

Pitfall 4: Forgetting X is a Function

Don't write "X = 5" when you mean "X(ω) = 5" or "X takes the value 5." X is a function, not a number! We often abuse notation, but keep the function nature in mind.

Test Your Understanding

Loading interactive demo...

Summary

Key Takeaways

A random variable is a function X: Ω → ℝ that assigns real numbers to outcomes in the sample space.
The range of X is the set of all possible values X can take— this is different from the sample space Ω!
A random variable is discrete if its range is countable(finite or countably infinite).
X is a deterministic function. The randomness comes from not knowing which outcome ω will occur.
In ML/AI, discrete RVs appear as: classification outputs, token predictions, discrete actions, attention positions.
Understanding discrete RVs is essential for working with probability distributions, loss functions, and sampling strategies.

Looking Ahead: Now that we understand what discrete random variables are, the next section explores their probability mass functions (PMFs)— the mathematical tool that describes how likely each value is to occur.