Probability Basics

Why Probability Matters for Neural Networks

In the previous sections, we learned how vectors and matrices transform data (Section 1) and how derivatives measure change (Section 2). Now we add the third essential ingredient:probability \u2014 the mathematics of uncertainty.

Neural networks are, at their core, probability machines. When a classifier looks at an image and outputs \u201C85% cat, 10% dog, 5% bird,\u201D it is producing a probability distribution. When we train the network, we measure how far its predicted distribution is from the true answer using a probability-based metric calledcross-entropy. When we initialize weights, we draw them from aGaussian distribution. Probability is woven into every aspect of deep learning.

The Big Picture: This section builds a straight path from basic probability concepts to the cross-entropy loss function \u2014 the single most important equation in classification neural networks. By the end, you will understand not just how to compute the loss, but why it takes the form it does, rooted in information theory and maximum likelihood estimation.

Here is how probability concepts connect to neural networks:

Random Variables and Probability Distributions

A random variable is a variable whose value is determined by a random process. It maps each outcome of a random experiment to a number. For example, when you roll a die, the random variable $X$ takes the value of whichever face lands up: 1, 2, 3, 4, 5, or 6.

A probability distribution describes how likely each value of the random variable is. It assigns a probability $P(X = k)$ to each possible outcome $k$. Two rules must always hold:

1. Every probability is between 0 and 1: $0 \leq P(X = k) \leq 1$
2. All probabilities sum to 1: $\sum_k P(X = k) = 1$ (something must happen)

Discrete vs. Continuous

Random variables come in two flavors:

• Discrete: Takes a finite or countable set of values. Examples: die roll (1\u20136), coin flip (0 or 1), a classifier's output class (cat, dog, bird). The distribution is described by a probability mass function (PMF): $P(X = k)$ gives the probability of each specific value.
• Continuous: Takes any value in a range. Examples: a person's height, a neural network weight, the output of a regression model. The distribution is described by a probability density function (PDF): $f(x)$. The probability of falling in an interval $[a, b]$ is the area under the curve: $P(a \leq X \leq b) = \int_a^b f(x)\,dx$.

Why This Matters: A neural network classifier outputs a discrete probability distribution: one probability per class. This is a PMF \u2014 and softmax ensures it sums to 1. Neural network weights, on the other hand, are continuous random variables during initialization, drawn from a Gaussian PDF.

Expected Value and Variance

Two numbers summarize the most important properties of any distribution: where it's centered (expected value) and how spread out it is (variance).

Expected Value (Mean)

The expected value $E[X]$ is the probability-weighted average of all possible outcomes. It answers: \u201CIf I repeated this experiment infinitely many times, what would the average outcome be?\u201D

For a discrete random variable:

$E[X] = \sum_k k \cdot P(X = k)$

For example, a fair die: $E[X] = 1 \cdot \frac{1}{6} + 2 \cdot \frac{1}{6} + 3 \cdot \frac{1}{6} + 4 \cdot \frac{1}{6} + 5 \cdot \frac{1}{6} + 6 \cdot \frac{1}{6} = 3.5$. Note that 3.5 is not a value the die can actually show \u2014 the expected value doesn't have to be a possible outcome.

Variance

Variance $\text{Var}[X]$ measures how spread out the distribution is around the mean. It is the expected value of the squared deviations from the mean:

$\text{Var}[X] = E[(X - \mu)^2] = \sum_k (k - \mu)^2 \cdot P(X = k)$

The standard deviation $\sigma = \sqrt{\text{Var}[X]}$ is the square root of the variance and has the same units as $X$, making it more interpretable: \u201Ctypically, outcomes are about $\sigma$ away from the mean.\u201D

Neural Network Connection: When training a neural network, the loss function computes the expected loss over the training data:$L = \frac{1}{N} \sum_{i=1}^{N} \ell(f(x_i), y_i)$ \u2014 this is a Monte Carlo estimate of $E[\ell]$. Variance matters too: high-variance gradients (noisy) slow training. Batch normalization reduces internal variance, and techniques like gradient clipping control extreme gradient values.

Let's compute expected value and variance in Python:

1import numpy as np
2
3# A discrete random variable: fair 6-sided die
4outcomes = np.array([1, 2, 3, 4, 5, 6])
5probabilities = np.array([1/6, 1/6, 1/6, 1/6, 1/6, 1/6])
6
7print("Outcomes:", outcomes)
8print("Probabilities:", np.round(probabilities, 4))
9print("Sum of probabilities:", probabilities.sum())
10
11# Expected value: E[X] = sum(x * P(x))
12expected_value = np.sum(outcomes * probabilities)
13print(f"\nE[X] = {expected_value:.4f}")
14
15# Variance: Var[X] = E[(X - mu)^2]
16variance = np.sum((outcomes - expected_value)**2 * probabilities)
17print(f"Var[X] = {variance:.4f}")
18
19# Standard deviation: sqrt(Var[X])
20std_dev = np.sqrt(variance)
21print(f"Std[X] = {std_dev:.4f}")
22
23# Simulate: roll the die 10,000 times
24np.random.seed(42)
25rolls = np.random.randint(1, 7, size=10000)
26print(f"\nSimulated E[X] = {np.mean(rolls):.4f}")
27print(f"Simulated Var[X] = {np.var(rolls):.4f}")

The Gaussian Distribution

The Gaussian distribution (also called the normal distribution) is the most important continuous distribution in all of machine learning. Its probability density function is the famous \u201Cbell curve\u201D:

$f(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\!\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$

This formula has two parameters:

• $\mu$ (mu): The mean \u2014 the center of the bell curve. The distribution is perfectly symmetric around $\mu$.
• $\sigma$ (sigma): The standard deviation \u2014 controls the width. Small $\sigma$ means a narrow, peaked bell; large $\sigma$ means a wide, flat bell.

Why the Gaussian is Everywhere

The Gaussian distribution appears throughout neural networks for deep mathematical reasons:

1. Central Limit Theorem: The sum of many independent random variables converges to a Gaussian, regardless of their individual distributions. Since a neuron computes $z = \sum_i w_i x_i + b$ (a sum of many terms), its pre-activation output tends to be approximately Gaussian.
2. Weight Initialization: We initialize neural network weights by drawing from a Gaussian distribution. Xavier initialization uses $\mathcal{N}(0, 1/n)$ and He initialization uses $\mathcal{N}(0, 2/n)$ where $n$ is the number of input neurons.
3. Batch Normalization: Standardizes activations to have $\mu = 0$ and $\sigma = 1$, effectively making them follow a standard Gaussian. This stabilizes training dramatically.
4. Regression Loss: The mean squared error (MSE) loss is equivalent to maximum likelihood estimation under the assumption that errors are Gaussian-distributed.

The 68-95-99.7 Rule

A powerful rule of thumb for any Gaussian distribution: approximately 68% of values fall within $\pm 1\sigma$ of the mean, 95% within $\pm 2\sigma$, and 99.7% within $\pm 3\sigma$. Anything beyond $3\sigma$ is extremely rare.

Explore the Gaussian distribution interactively by adjusting the mean and standard deviation:

Now let's implement the Gaussian PDF from scratch and verify these properties:

  P(x= -2.0) = 0.053991
  P(x= -1.0) = 0.241971
  P(x=  0.0) = 0.398942
  P(x=  1.0) = 0.241971
  P(x=  2.0) = 0.053991

1import numpy as np
2
3# The Gaussian (Normal) PDF
4def gaussian_pdf(x, mu, sigma):
5    coeff = 1 / (sigma * np.sqrt(2 * np.pi))
6    exponent = -((x - mu)**2) / (2 * sigma**2)
7    return coeff * np.exp(exponent)
8
9# Standard normal: mu=0, sigma=1
10mu, sigma = 0.0, 1.0
11
12# Evaluate PDF at specific points
13print("Standard Normal PDF values:")
14for x in [-2.0, -1.0, 0.0, 1.0, 2.0]:
15    p = gaussian_pdf(x, mu, sigma)
16    print(f"  P(x={x:5.1f}) = {p:.6f}")
17
18# The 68-95-99.7 rule
19print("\n68-95-99.7 Rule:")
20print(f"  P(-1 < X < 1) = 68.27%")
21print(f"  P(-2 < X < 2) = 95.45%")
22print(f"  P(-3 < X < 3) = 99.73%")
23
24# Generate samples
25np.random.seed(42)
26samples = np.random.normal(mu, sigma, size=5)
27print(f"\n5 samples: {np.round(samples, 4)}")
28print(f"Sample mean: {np.mean(samples):.4f}")
29print(f"Sample std:  {np.std(samples):.4f}")

Conditional Probability and Bayes' Theorem

Conditional probability is the probability of an event given that another event has already occurred. It is written $P(A \mid B)$ and read as \u201Cthe probability of A given B.\u201D

The formula is: $P(A \mid B) = \frac{P(A \cap B)}{P(B)}$

This is exactly what neural network classifiers do: they compute $P(\text{class} \mid \text{input})$ \u2014 the probability of each class given the input data. When a model sees an image and outputs \u201C85% cat,\u201D it is computing $P(\text{cat} \mid \text{image}) = 0.85$.

Bayes' Theorem

Bayes' theorem lets us invert conditional probabilities. If we know $P(B \mid A)$ (the likelihood), we can compute $P(A \mid B)$ (the posterior):

$P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}$

The terms have specific names:

Let's see Bayes' theorem in action with a famous counterintuitive example:

1import numpy as np
2
3# Medical test: disease with 1% prevalence
4# Test accuracy: 99% true positive, 5% false positive
5
6# Prior probability
7p_disease = 0.01        # 1% of population has disease
8p_no_disease = 0.99     # 99% are healthy
9
10# Likelihood: how accurate is the test?
11p_pos_given_disease = 0.99    # true positive rate
12p_pos_given_no_disease = 0.05 # false positive rate
13
14# Total probability of testing positive (law of total probability)
15p_positive = (p_pos_given_disease * p_disease +
16              p_pos_given_no_disease * p_no_disease)
17
18print(f"P(positive) = {p_positive:.4f}")
19
20# Bayes' theorem: P(disease | positive)
21p_disease_given_pos = (p_pos_given_disease * p_disease) / p_positive
22
23print(f"P(disease | positive) = {p_disease_given_pos:.4f}")
24print(f"That is only {p_disease_given_pos*100:.2f}%!")
25
26# Even with a 99% accurate test, if the disease is rare,
27# most positive results are false positives!
28print(f"\nFalse positive contribution: {p_pos_given_no_disease * p_no_disease:.4f}")
29print(f"True positive contribution:  {p_pos_given_disease * p_disease:.4f}")

Key Insight: Even a 99% accurate test gives only a 16.7% chance of disease when the base rate is 1%. This is the base rate fallacy. In neural networks, the same principle applies: if a class is very rare in the training data, the model may achieve high accuracy by simply predicting the majority class. This is why we use balanced accuracy, precision/recall, and F1 score instead of raw accuracy for imbalanced datasets.

From Logits to Probabilities: The Softmax Function

Neural network classification layers output raw numbers called logits \u2014 they can be any real number (positive, negative, or zero) and don't form a valid probability distribution. The softmax function converts these raw scores into probabilities:

$\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$

Softmax does three things simultaneously:

1. Makes all values positive: The exponential $e^{z_i}$ is always positive, even for negative logits. This ensures no negative \u201Cprobabilities.\u201D
2. Normalizes to sum to 1: Dividing by the sum $\sum e^{z_j}$ guarantees the outputs form a valid probability distribution.
3. Preserves ordering: Larger logits always get larger probabilities. The class with the highest logit gets the highest probability.

The Numerical Stability Trick

In practice, we subtract the maximum logit before computing softmax: $\text{softmax}(z_i) = \text{softmax}(z_i - \max(z))$. This prevents $e^{\text{large number}} = \infty$ (overflow) without changing the result. The proof is elegant: the $\max(z)$ constant cancels in the numerator and denominator.

Temperature Scaling

Dividing logits by a temperature parameter $T$ before softmax controls the \u201Csharpness\u201D of the output distribution:$\text{softmax}(z_i / T)$.

• $T \to 0$: Distribution becomes one-hot (all probability on the argmax). The model is maximally confident.
• $T = 1$: Standard softmax. The default.
• $T \to \infty$: Distribution becomes uniform ($1/K$ for each class). The model expresses maximum uncertainty.

Temperature scaling is used in knowledge distillation (training a small model from a large model's soft predictions) and language model sampling (higher temperature = more creative text generation).

Experiment with softmax interactively \u2014 adjust the logits and temperature to see how they affect the output probabilities:

Now let's implement softmax from scratch with the numerical stability trick and temperature:

  cat  : logit=  2.0 -> P=0.6590
  dog  : logit=  1.0 -> P=0.2424
  bird : logit=  0.1 -> P=0.0986

  T=0.5: [0.8638 0.1169 0.0193]
  T=1.0: [0.659  0.2424 0.0986]
  T=2.0: [0.5017 0.3043 0.194 ]
  T=5.0: [0.3996 0.3272 0.2733]

1import numpy as np
2
3def softmax(logits, temperature=1.0):
4    """Convert raw scores to probabilities."""
5    # Scale by temperature
6    scaled = logits / temperature
7
8    # Subtract max for numerical stability
9    shifted = scaled - np.max(scaled)
10
11    # Exponentiate: all values become positive
12    exp_vals = np.exp(shifted)
13
14    # Normalize: divide by sum so they sum to 1
15    probs = exp_vals / np.sum(exp_vals)
16    return probs
17
18# Network outputs (logits) for 3 classes
19logits = np.array([2.0, 1.0, 0.1])
20labels = ["cat", "dog", "bird"]
21
22probs = softmax(logits)
23print("Class probabilities:")
24for label, logit, prob in zip(labels, logits, probs):
25    print(f"  {label:5s}: logit={logit:5.1f} -> P={prob:.4f}")
26print(f"  Sum = {probs.sum():.4f}")
27
28# Temperature effect: controls "confidence"
29print("\nTemperature effect:")
30for T in [0.5, 1.0, 2.0, 5.0]:
31    p = softmax(logits, temperature=T)
32    print(f"  T={T:.1f}: {np.round(p, 4)}")

Cross-Entropy: Measuring Prediction Quality

Now we arrive at the most important equation in classification neural networks:cross-entropy loss. It measures how different the model's predicted probability distribution $q$ is from the true distribution $p$:

$H(p, q) = -\sum_{k=1}^{K} p(k) \cdot \log q(k)$

Since the true distribution $p$ is one-hot (only one class is correct), cross-entropy simplifies dramatically:

$H(p, q) = -\log q(y_{\text{true}})$

This is just the negative log probability of the correct class. The properties are beautiful:

Notice the asymmetric penalty: going from 0.99 to 0.7 adds only 0.35 to the loss, but going from 0.1 to 0.01 adds 2.31. Cross-entropy punishes confident wrong answers exponentially more than slightly uncertain correct answers. This creates a strong gradient signal to fix wrong predictions.

Information Theory Perspective

Cross-entropy comes from information theory. The entropy of a distribution $p$ is $H(p) = -\sum p(k) \log p(k)$ \u2014 the minimum average number of bits needed to encode events from $p$. The cross-entropy $H(p, q)$ is the average number of bits needed when you use the encoding optimized for $q$ but the data actually comes from $p$. The difference $H(p, q) - H(p)$ is the KL divergence $D_{KL}(p \| q)$ \u2014 the \u201Cextra bits\u201D wasted by using the wrong distribution. Since $H(p)$ is constant during training (the true labels don't change), minimizing cross-entropy is equivalent to minimizing KL divergence.

Explore cross-entropy interactively \u2014 adjust the predicted probabilities and see how the loss changes:

Let's implement cross-entropy and see how different predictions affect the loss:

1import numpy as np
2
3def cross_entropy(true_probs, pred_probs):
4    """H(p, q) = -sum(p(x) * log(q(x)))"""
5    eps = 1e-15  # prevent log(0)
6    return -np.sum(true_probs * np.log(pred_probs + eps))
7
8# True label: "cat" (one-hot encoded)
9true = np.array([1.0, 0.0, 0.0])
10labels = ["cat", "dog", "bird"]
11
12# Scenario 1: Good prediction
13good_pred = np.array([0.7, 0.2, 0.1])
14loss_good = cross_entropy(true, good_pred)
15print(f"Good prediction {good_pred}:")
16print(f"  Loss = -log(0.7) = {loss_good:.4f}")
17
18# Scenario 2: Bad prediction
19bad_pred = np.array([0.1, 0.6, 0.3])
20loss_bad = cross_entropy(true, bad_pred)
21print(f"\nBad prediction {bad_pred}:")
22print(f"  Loss = -log(0.1) = {loss_bad:.4f}")
23
24# Scenario 3: Near-perfect prediction
25perfect_pred = np.array([0.99, 0.005, 0.005])
26loss_perfect = cross_entropy(true, perfect_pred)
27print(f"\nPerfect prediction {perfect_pred}:")
28print(f"  Loss = -log(0.99) = {loss_perfect:.4f}")
29
30# Comparison
31print("\n--- Summary ---")
32print(f"Perfect:  {loss_perfect:.4f} (lowest)")
33print(f"Good:     {loss_good:.4f}")
34print(f"Bad:      {loss_bad:.4f} (highest)")

Maximum Likelihood: Why Cross-Entropy Works

Why is cross-entropy the loss function for classification? The answer comes from Maximum Likelihood Estimation (MLE) \u2014 the most principled way to fit a model to data.

The Likelihood Function

Given a dataset of $N$ examples $\{(x_1, y_1), \ldots, (x_N, y_N)\}$, the likelihood is the probability of observing all the true labels given the model's predictions. Assuming independence:

$\mathcal{L}(\theta) = \prod_{i=1}^{N} P_{\theta}(y_i \mid x_i)$

Here $P_{\theta}(y_i \mid x_i)$ is the probability that the model (with parameters $\theta$) assigns to the correct class $y_i$ for input $x_i$. We want to find $\theta$ that maximizes this likelihood \u2014 make the data as probable as possible under our model.

From Likelihood to Cross-Entropy

Products are numerically unstable and hard to optimize. Taking the log converts the product to a sum (since $\log(a \cdot b) = \log a + \log b$):

$\log \mathcal{L}(\theta) = \sum_{i=1}^{N} \log P_{\theta}(y_i \mid x_i)$

Maximizing this log-likelihood is equivalent to minimizing its negative:

$-\frac{1}{N} \sum_{i=1}^{N} \log P_{\theta}(y_i \mid x_i)$

And this is exactly the cross-entropy loss! Each term $-\log P_{\theta}(y_i \mid x_i)$ is the negative log probability of the correct class \u2014 exactly what cross-entropy computes.

The Connection: Cross-entropy loss is not an arbitrary choice. It is the mathematically principled consequence of maximum likelihood estimation. When you minimize cross-entropy during training, you are maximizing the probability that the model assigns to the correct labels \u2014 you are finding the parameters that make the observed data most likely.

Why Not Use MSE for Classification?

You might wonder: why not use mean squared error (MSE) for classification? The answer is both theoretical and practical:

1. MSE assumes Gaussian errors. MSE is the maximum likelihood estimator when errors follow a Gaussian distribution. But classification labels are categorical (one-hot), not Gaussian \u2014 so MSE is the wrong probabilistic model.
2. Gradient saturation. With softmax + MSE, the gradient can vanish when the prediction is very wrong (the sigmoid-like saturation effect). With softmax + cross-entropy, the gradient is simply $\hat{y} - y$ (predicted minus true) \u2014 no saturation, always a clear learning signal.
3. Sharper convergence. Cross-entropy creates steeper gradients for confident wrong predictions. If the model says 99% dog when the answer is cat, cross-entropy produces a massive gradient. MSE would produce a relatively gentle gradient for the same error.

Probability in PyTorch

PyTorch provides optimized, numerically stable implementations of softmax and cross-entropy that you should always use in practice (never implement them from scratch in production code). The key function is $\texttt{F.cross\_entropy}$, which combines softmax and negative log-likelihood in a single numerically stable operation.

Important: PyTorch's $\texttt{F.cross\_entropy}$ takes raw logits, not probabilities. It applies softmax internally. If you pass probabilities that have already been through softmax, you'll get wrong results (double softmax).

Let's see the complete PyTorch pipeline: logits \u2192 softmax \u2192 cross-entropy \u2192 gradients:

1import torch
2import torch.nn.functional as F
3
4# Network outputs (raw logits, NOT probabilities)
5logits = torch.tensor([[2.0, 1.0, 0.1]])
6target = torch.tensor([0])  # class index for "cat"
7
8# PyTorch combines softmax + cross-entropy in one step
9loss = F.cross_entropy(logits, target)
10print(f"Cross-entropy loss: {loss.item():.4f}")
11
12# Manual verification: softmax then negative log-likelihood
13probs = F.softmax(logits, dim=-1)
14print(f"Softmax probabilities: {probs.detach().numpy()}")
15
16manual_loss = -torch.log(probs[0, target[0]])
17print(f"Manual -log(p[cat]): {manual_loss.item():.4f}")
18
19# Gradient: how should logits change to reduce loss?
20logits_grad = torch.tensor([[2.0, 1.0, 0.1]], requires_grad=True)
21loss = F.cross_entropy(logits_grad, target)
22loss.backward()
23print(f"\nGradients: {logits_grad.grad.numpy()}")
24print(f"Gradient = softmax - one_hot")

The Beautiful Gradient

The gradient of cross-entropy loss with respect to the logits has a remarkably simple form: $\frac{\partial L}{\partial z_i} = \hat{y}_i - y_i$, where $\hat{y}$ is the softmax output and $y$ is the one-hot target. In our example:

The gradient is simply the prediction error: how far each predicted probability is from the true value. This elegance is no accident \u2014 it is a direct consequence of the mathematical relationship between softmax and cross-entropy being dual to each other.

Summary and What's Next

We've built a complete path from basic probability to the cross-entropy loss function. Here's the chain of ideas:

1. Random variables and distributions formalize uncertainty \u2014 each outcome gets a probability, and probabilities must sum to 1.
2. Expected value and variance summarize distributions with two numbers: where it's centered and how spread out it is.
3. The Gaussian distribution appears everywhere in neural networks: weight initialization, batch normalization, and the central limit theorem.
4. Bayes' theorem lets us reason about $P(\text{class} \mid \text{data})$ \u2014 exactly what classifiers compute \u2014 and warns us about base rate effects.
5. Softmax converts raw network outputs (logits) into a valid probability distribution, with temperature controlling confidence.
6. Cross-entropy measures how different two distributions are. For one-hot labels, it simplifies to $-\log P(\text{correct class})$.
7. Maximum likelihood estimation proves that cross-entropy is the mathematically principled loss for classification \u2014 not an arbitrary choice.

What's Next: In the next section, we cover the chain rule \u2014 the mathematical engine that powers backpropagation. The chain rule lets us compute how the loss changes with respect to every single weight in the network, enabling gradient descent to work on deep networks with millions of parameters.