Backpropagation Derivation

Learning Objectives

By the end of this section, you will be able to:

1. Understand the credit assignment problem and why we need an efficient algorithm to compute gradients
2. Apply the chain rule to composite functions and understand how derivatives propagate backward
3. Trace gradient flow through computation graphs and handle multiple paths
4. Derive the four fundamental equations of backpropagation from first principles
5. Express backpropagation in matrix form for efficient computation
6. Connect the mathematical derivation to practical implementation

Why This Matters: Backpropagation is the algorithm that makes deep learning possible. Without it, training neural networks with millions of parameters would be computationally infeasible. Understanding its derivation reveals why neural networks can learn and demystifies the "black box" of gradient computation.

The Credit Assignment Problem

Imagine you're training a neural network with 1000 weights. When the network makes a prediction error, you want to update each weight to reduce this error. But how much should each weight change? This is the credit assignment problem: assigning credit (or blame) to each parameter for the network's output.

The Naive Approach: Numerical Gradients

One approach is to compute gradients numerically. For each weight $w_i$, we could estimate:

This requires two forward passes per weight. For a network with $n$ weights, we need $2n$ forward passes—completely impractical for modern networks with millions of parameters.

The Key Insight

Backpropagation exploits the structure of neural networks—the fact that they are composed of simple, differentiable operations connected in a graph. By applying the chain rule systematically, we can compute all gradients in a single backward pass.

Chain Rule: The Mathematical Foundation

Backpropagation is fundamentally an application of the chain rule from calculus. Let's review this essential concept before diving into neural networks.

Single Variable Chain Rule

For a composite function $f(g(x))$, the chain rule states:

In words: the derivative of the composite function equals the derivative of the outer function (evaluated at the inner function) times the derivative of the inner function.

Multivariate Chain Rule

For functions of multiple variables, if $z = f(u, v)$ where $u = g(x, y)$ and $v = h(x, y)$, then:

When a variable influences the output through multiple paths, we sum the contributions from each path.

The Backpropagation Principle: Multiply local gradients along each path, then sum across all paths from a variable to the output.

Interactive: Chain Rule Explorer

Explore how the chain rule works for composite functions. Adjust the input value to see how derivatives are computed step by step.

Gradient Flow in Computation Graphs

Neural networks can be represented as computation graphs where nodes represent operations and edges represent data flow. Understanding how gradients flow through these graphs is key to understanding backpropagation.

Forward and Backward Passes

Every computation graph supports two passes:

1. Forward Pass: Compute the output by evaluating nodes in topological order (inputs → output)
2. Backward Pass: Compute gradients by propagating errors in reverse topological order (output → inputs)

Local Gradients

Each operation in the graph has a local gradient—the derivative of its output with respect to each of its inputs. During the backward pass, we:

1. Receive the gradient flowing into this node (from downstream)
2. Multiply by the local gradient
3. Pass the result upstream

The Sum Rule for Multiple Paths

When a variable is used multiple times (has multiple outgoing edges), the gradients from each path are summed:

Interactive: Gradient Flow Demo

Watch how gradients flow backward through a computation graph. Notice how gradients are multiplied along paths and summed when paths merge.

Deriving Backpropagation

Now let's derive backpropagation for neural networks step by step, starting from the simplest case and building up complexity.

Single Neuron Case

Consider a single neuron with input $x$, weight $w$, bias $b$, and activation function $\sigma$:

To find $\partial L / \partial w$, we apply the chain rule:

Computing each term:

• $\partial L / \partial a = a - y$ (derivative of squared error)
• $\partial a / \partial z = \sigma'(z)$ (derivative of activation function)
• $\partial z / \partial w = x$ (derivative of linear combination)

Therefore:

Single Layer Case

Now consider a layer with multiple neurons. For the $j$-th neuron with inputs $\mathbf{x} = (x_1, \ldots, x_n)$:

The gradient with respect to weight $w_{jk}$:

We can define the error term (or delta) for neuron $j$:

Then the weight gradient becomes simply:

Multi-Layer Network

For a network with layers $l = 1, 2, \ldots, L$, we need to propagate error terms backward through the layers. Consider the error term for neuron $j$ in layer $l$:

By the chain rule, this error depends on how neuron $j$ in layer $l$ affects all neurons in layer $l+1$:

where:

This gives us the backward propagation equation:

The Propagation Insight: The error at a hidden neuron is the weighted sum of errors from all neurons it connects to in the next layer, scaled by the activation derivative. Errors flow backward through the same weights used in the forward pass!

The Four Fundamental Equations

The entire backpropagation algorithm can be summarized in four equations. Click each equation to expand its intuition:

Equation 1: Output Layer Error

Where $\nabla_a C$ is the gradient of the cost with respect to the output activations, and $\odot$ denotes element-wise multiplication. For mean squared error: $\nabla_a C = (a^{(L)} - y)$.

Equation 2: Error Backpropagation

This propagates the error from layer $l+1$ back to layer $l$. The transposed weight matrix rotates the error signal back through the connections.

Equation 3: Weight Gradients

The weight gradient is an outer product of the error at a layer and the activations feeding into it. This makes intuitive sense: weights connecting active neurons to neurons with large errors should change the most.

Equation 4: Bias Gradients

The bias gradient equals the error term directly. Biases are like weights with a fixed input of 1.

Interactive: Backpropagation Step-by-Step

Watch backpropagation unfold step by step in a simple two-layer network. Observe how gradients are computed and flow backward from the loss to each weight.

Matrix Formulation

For efficient computation, we express backpropagation in matrix form. This enables vectorized operations and GPU acceleration.

Forward Pass (Vectorized)

For a batch of $m$ examples with input matrix $X \in \mathbb{R}^{n \times m}$:

Where:

• $W^{(l)} \in \mathbb{R}^{n_l \times n_{l-1}}$ is the weight matrix
• $A^{(l)} \in \mathbb{R}^{n_l \times m}$ is the activation matrix (each column is one example)
• $b^{(l)} \in \mathbb{R}^{n_l \times 1}$ is broadcast across examples

Backward Pass (Vectorized)

The error terms and gradients for a batch:

Implementation Preview

Here's a preview of how backpropagation is implemented in code. We'll build the complete implementation in the next section.

If layer l+1 has error [0.1, -0.2] and W^T maps this to layer l, we get the 'responsibility' of each hidden neuron for the downstream error.

1import numpy as np
2
3class BackpropNetwork:
4    """Neural network with backpropagation implemented from scratch."""
5
6    def __init__(self, layer_sizes):
7        """Initialize network with given architecture.
8
9        Args:
10            layer_sizes: List of layer sizes, e.g., [784, 128, 10]
11        """
12        self.num_layers = len(layer_sizes)
13        self.sizes = layer_sizes
14
15        # Initialize weights with Xavier/He initialization
16        self.weights = [
17            np.random.randn(y, x) * np.sqrt(2.0 / x)
18            for x, y in zip(layer_sizes[:-1], layer_sizes[1:])
19        ]
20        self.biases = [np.zeros((y, 1)) for y in layer_sizes[1:]]
21
22    def sigmoid(self, z):
23        """Sigmoid activation function."""
24        return 1.0 / (1.0 + np.exp(-np.clip(z, -500, 500)))
25
26    def sigmoid_derivative(self, a):
27        """Derivative of sigmoid: σ'(z) = a(1-a) where a = σ(z)."""
28        return a * (1 - a)
29
30    def forward(self, X):
31        """Forward pass: compute activations for all layers.
32
33        Args:
34            X: Input data, shape (n_features, batch_size)
35
36        Returns:
37            Tuple of (activations, pre_activations) for all layers
38        """
39        activations = [X]
40        zs = []  # Pre-activations
41
42        a = X
43        for w, b in zip(self.weights, self.biases):
44            z = np.dot(w, a) + b  # Linear transformation
45            zs.append(z)
46            a = self.sigmoid(z)   # Activation
47            activations.append(a)
48
49        return activations, zs
50
51    def backward(self, X, y):
52        """Backward pass: compute gradients using backpropagation.
53
54        Args:
55            X: Input data, shape (n_features, batch_size)
56            y: True labels, shape (n_outputs, batch_size)
57
58        Returns:
59            Tuple of (weight_gradients, bias_gradients)
60        """
61        m = X.shape[1]  # Batch size
62        grad_w = [np.zeros_like(w) for w in self.weights]
63        grad_b = [np.zeros_like(b) for b in self.biases]
64
65        # Forward pass (store all activations)
66        activations, zs = self.forward(X)
67
68        # EQUATION 1: Output layer error
69        # δ^L = ∇_a C ⊙ σ'(z^L)
70        delta = (activations[-1] - y) * self.sigmoid_derivative(activations[-1])
71
72        # EQUATION 3 & 4: Gradients for last layer
73        grad_w[-1] = np.dot(delta, activations[-2].T) / m
74        grad_b[-1] = np.mean(delta, axis=1, keepdims=True)
75
76        # Backpropagate through hidden layers
77        for l in range(2, self.num_layers):
78            # EQUATION 2: Backpropagate error
79            # δ^l = (W^{l+1})^T δ^{l+1} ⊙ σ'(z^l)
80            delta = np.dot(self.weights[-l+1].T, delta) * \
81                    self.sigmoid_derivative(activations[-l])
82
83            # EQUATION 3 & 4: Gradients for this layer
84            grad_w[-l] = np.dot(delta, activations[-l-1].T) / m
85            grad_b[-l] = np.mean(delta, axis=1, keepdims=True)
86
87        return grad_w, grad_b
88
89    def compute_loss(self, X, y):
90        """Compute mean squared error loss."""
91        activations, _ = self.forward(X)
92        predictions = activations[-1]
93        return 0.5 * np.mean((predictions - y) ** 2)
94
95# Example usage
96network = BackpropNetwork([2, 3, 1])  # 2 inputs, 3 hidden, 1 output
97
98# Random example
99X = np.random.randn(2, 5)  # 5 examples
100y = np.random.randn(1, 5)
101
102# Compute gradients
103grad_w, grad_b = network.backward(X, y)
104
105print("Weight gradients shapes:", [g.shape for g in grad_w])
106print("Bias gradients shapes:", [g.shape for g in grad_b])

Common Misconceptions

Let's address some common misconceptions about backpropagation:

Misconception 1: "Backpropagation is a learning algorithm"

Reality: Backpropagation is an algorithm for computing gradients efficiently. The learning algorithm is gradient descent (or its variants like Adam, SGD with momentum, etc.). Backpropagation provides the gradients that gradient descent uses to update weights.

Misconception 2: "Gradients flow through the network during backprop"

Reality: This is a useful mental model, but physically, backpropagation is just matrix multiplications computed in reverse order. The "flow" is a metaphor for the computational dependency structure.

Misconception 3: "We need to store all forward pass values"

Partially true: Standard backprop requires storing activations from the forward pass. However, techniques like gradient checkpointing can trade computation for memory by recomputing some activations during the backward pass.

Misconception 4: "Backpropagation doesn't work with ReLU because it's not differentiable at 0"

Reality: While ReLU isn't differentiable at exactly 0, we simply define the derivative as 0 (or sometimes 1) at that point. This works well in practice because hitting exactly 0 is rare with floating-point numbers.

Summary

Backpropagation is the efficient algorithm that makes training deep neural networks possible. Let's review the key concepts:

The Algorithm in Three Steps

1. Forward Pass: Compute and store all activations from input to output
2. Compute Output Error: Calculate $\delta^{(L)}$ from the loss gradient and activation derivative
3. Backward Pass: Propagate errors backward, computing weight and bias gradients at each layer

The Big Picture: Backpropagation is simply the chain rule applied systematically to a computation graph. Its elegance lies in computing ALL gradients with just one additional pass through the network, enabling the training of networks with millions of parameters.

Exercises

Conceptual Questions

1. Explain why the gradient with respect to a weight depends on both the error at that neuron AND the activation feeding into it. Give an intuitive explanation.
2. In the error backpropagation equation, why do we use the transpose of the weight matrix?
3. What happens to the gradient flow if a neuron's activation is in the saturated region of the sigmoid (close to 0 or 1)? How does this relate to the vanishing gradient problem?
4. Why is it important that the forward and backward pass have approximately the same computational cost? What would happen if backward was much more expensive?

Mathematical Derivations

1. Derive the backpropagation equations for a network using the cross-entropy loss instead of MSE. What changes in the output error term?
2. Derive the gradient of the loss with respect to a bias term. Show that it equals the error term at that neuron.
3. For a network with ReLU activations, write out the backward pass equations. What is σ'(z) for ReLU?

Coding Exercises

1. Gradient Checking: Implement numerical gradient computation and use it to verify that your backpropagation implementation is correct. Compare the analytical gradient to the numerical gradient for each weight.
2. Gradient Flow Analysis: Modify the BackpropNetwork class to track and visualize the magnitude of gradients at each layer during training. Plot how gradient magnitudes change over training epochs.
3. Batch Processing: Extend the implementation to handle batches of arbitrary size efficiently using NumPy broadcasting.

Challenge Exercise

Implement Gradient Checkpointing: Modify the BackpropNetwork class to use gradient checkpointing—instead of storing all activations, only store activations at every k-th layer and recompute the others during the backward pass. Measure the memory vs. computation trade-off.

In the next section, we'll implement backpropagation from scratch in pure Python/NumPy, and then see how PyTorch's autograd system does all of this automatically.