The Vanishing Gradient Problem

The Memory Problem

In the last section we built Backpropagation Through Time and watched an RNN actually learn. It worked beautifully — on sentences of four words. But if we had pushed that same network on a sentence of fifty words and asked it to connect a subject to a verb far, far away, we would have watched it fail silently. No error. No crash. The weights just would not move in the right direction. The network would behave, stubbornly, as if the first half of the sequence did not exist.

This is the vanishing gradient problem, and it is not a bug. It is a mathematical consequence of how RNNs share weights across time. In this section we are going to see, with full rigor and with numbers you can click through line by line, exactly why the signal dies on its way back through time, and why this single difficulty held back sequence modeling for nearly a decade — until LSTM and GRU stepped in.

The one-sentence idea. A vanilla RNN multiplies its gradient by the same matrix $W_{hh}$ at every step backward. A product of many matrices with spectral radius less than 1 collapses to zero exponentially; a product with spectral radius greater than 1 explodes to infinity. There is almost no middle ground.

The Chain Rule Unrolled Through Time

Recall the RNN update from Chapter 16 Section 2:

$h_t = \tanh(W_{xh}\, x_t + W_{hh}\, h_{t-1} + b_h)$

We want to train this with gradient descent, so we need $\partial L / \partial W_{hh}$. The loss $L$ depends on the final hidden state $h_T$, and $h_T$ depends on $h_{T-1}$, which depends on $h_{T-2}$, all the way back to $h_1$. Applying the chain rule through this dependency chain gives

$\frac{\partial L}{\partial W_{hh}} = \sum_{t=1}^{T} \frac{\partial L}{\partial h_T} \cdot \frac{\partial h_T}{\partial h_t} \cdot \frac{\partial h_t}{\partial W_{hh}}.$

That sum looks innocent, but every single term contains the factor $\partial h_T / \partial h_t$ — a Jacobian that spans many timesteps. If even one of those factors is tiny, the corresponding term contributes almost nothing to the update of $W_{hh}$. If every one of them is tiny for early $t$, the network gets essentially no information about what happened at the start of the sequence.

The Jacobian Product: Where It All Goes Wrong

Expand $\partial h_T / \partial h_t$ using the chain rule again, one step at a time:

$\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} \frac{\partial h_k}{\partial h_{k-1}}.$

Each factor is the Jacobian of one RNN step. For the tanh recurrence above it equals

$\frac{\partial h_k}{\partial h_{k-1}} = \operatorname{diag}\!\bigl(1 - h_k^{\,2}\bigr)\; W_{hh}.$

So the matrix that transports the gradient from time $T$ back to time $t$ is a product of T - t copies of $\operatorname{diag}(1 - h_k^2)\, W_{hh}$. Taking norms and using submultiplicativity of matrix norms,

$\left\| \frac{\partial h_T}{\partial h_t} \right\| \leq \prod_{k=t+1}^{T} \left\| \operatorname{diag}(1-h_k^2) \right\| \; \left\| W_{hh} \right\|.$

The diagonal factor is bounded by $1$ because $\tanh' \leq 1$. So in the best case the bound collapses to

$\left\| \frac{\partial h_T}{\partial h_t} \right\| \leq \left\| W_{hh} \right\|^{\,T-t}.$

Two regimes jump out immediately:

• If $\|W_{hh}\| < 1$: the bound is $c^{T-t}$ with $c < 1$. The gradient shrinks exponentially in the sequence length. This is the vanishing regime.
• If $\|W_{hh}\| > 1$: the matrix $W_{hh}^{T-t}$ can grow exponentially. Unless tanh saturation happens to cancel it, gradients explode.

A Scalar Intuition: Why 0.5²⁰ Destroys Learning

All of the matrix mystery disappears once you look at a scalar version. Consider the one-dimensional recurrence $h_t = \tanh(w\, h_{t-1})$ with weight $w$. In the linear regime the local Jacobian is just $w$, and the full gradient over $T$ steps is

$\frac{\partial h_T}{\partial h_0} \approx w^{\,T}.$

Plug in real numbers. For a sentence of 20 words ($T = 20$):

This is the entire story in a single table. The vanishing gradient problem is not subtle: it is simply exponential decay. If you want the earliest word of a sentence to influence the last word's loss, every multiplicative factor along the chain must be very close to 1 — and random initialization plus tanh saturation conspire to make that almost impossible.

How Activation Functions Make It Worse

The bound $\|W_{hh}\|^{\,T-t}$ assumes the activation derivative sits at its maximum value of 1. In practice it does not. Each activation introduces its own shrinking factor:

Sigmoid is the most cruel: even if $\|W_{hh}\| = 1$, the per-step gradient is bounded by $0.25$. Over 20 steps that gives $0.25^{20} \approx 10^{-12}$. That is why modern RNN implementations almost always use tanh (or, inside LSTMs, use sigmoid only as a gate, never on the gradient highway).

Vanishing vs. Exploding: Two Faces of the Same Problem

Vanishing and exploding gradients are the same phenomenon with different signs in the exponent. Both arise because the gradient is a product, not a sum. Small differences in each factor compound multiplicatively.

Exploding gradients are usually fixed in 3 lines of code with torch.nn.utils.clip_grad_norm_ — the standard technique we already saw in Section 2. Vanishing gradients, however, cannot be clipped away. The signal is simply not there. To recover it, we need to change the architecture so that gradients can flow through a different, non-multiplicative path. That is exactly what LSTM and GRU accomplish.

Watch Gradients Flow Through Time

Before we run any code, let us see the phenomenon directly. Below is an interactive RNN where you can choose the activation, the weight scale ($\|W_{hh}\|$), and the sequence length. Watch how the earliest hidden state receives almost no signal at all once you push the sequence past eight or ten steps, and how sigmoid is far worse than tanh.

Python from Scratch: Watching Gradients Die

The surest way to believe the mathematics is to compute it. In this code block we trace the scalar gradient $\partial h_t / \partial h_0$ step by step for a one-dimensional RNN with weight $w = 0.5$. Every line has a click-to-reveal annotation showing the exact values flowing through the computer's memory: the hidden state, the local Jacobian at each step, and the cumulative product that IS the backpropagated gradient.

1import numpy as np
2
3# Scalar RNN: h_t = tanh(w * h_{t-1})
4# We track how the gradient dh_t/dh_0 shrinks at every step.
5def trace_gradient(w, T, h0=0.1):
6    h = h0
7    cumulative = 1.0
8    log = []
9    for t in range(1, T + 1):
10        h = np.tanh(w * h)
11        # Local Jacobian at step t: dh_t / dh_{t-1}
12        local = w * (1.0 - h ** 2)
13        # Full gradient dh_t / dh_0 is the product of local Jacobians
14        cumulative *= local
15        log.append((t, h, local, cumulative))
16    return log
17
18# Case 1: small weight -> vanishing gradient
19for t, h, local, cum in trace_gradient(w=0.5, T=10):
20    print(f"t={t:2d}  h={h:8.5f}  local={local:.5f}  cumprod={cum:.3e}")

Read the right-hand column of the output: cumprod. At $t=1$ it is 0.4988. At $t=10$ it is 9.7 × 10⁻⁴. That is a thousand-fold shrinkage in ten steps, and every step is the same operation — multiplying by roughly 0.5. No bug, no pathology; just the chain rule doing exactly what it is supposed to do.

PyTorch: Measuring Vanishing Gradients

Now let us scale up to a real RNN and use PyTorch's autograd to measure $\|\partial L / \partial h_t\|$ at every timestep. The key trick is retain_grad(), which forces PyTorch to keep the gradient on intermediate tensors so we can inspect them. We initialize $W_{hh}$ with spectral radius ≈ 0.5, pass 20 tiny inputs through the unrolled network, and let backward() do the rest.

        c0    c1    c2    c3
r0   0.5   0.0   0.0   0.0
r1   0.0   0.5   0.0   0.0
r2   0.0   0.0   0.5   0.0
r3   0.0   0.0   0.0   0.5

1import torch
2import torch.nn as nn
3
4torch.manual_seed(0)
5
6T = 20                  # sequence length
7hidden_size = 4
8input_size = 2
9
10# Recurrent weights chosen so spectral radius < 1 (vanishing regime)
11W_xh = nn.Linear(input_size, hidden_size, bias=False)
12W_hh = nn.Linear(hidden_size, hidden_size, bias=False)
13with torch.no_grad():
14    W_hh.weight.copy_(
15        0.5 * torch.eye(hidden_size)
16        + 0.05 * torch.randn(hidden_size, hidden_size)
17    )
18
19# Tiny random inputs so tanh stays in its linear regime
20x = 0.05 * torch.randn(T, input_size)
21
22# Build the unrolled RNN and retain gradients at each hidden state
23h = torch.zeros(hidden_size, requires_grad=True)
24hidden_states = [h]
25for t in range(T):
26    h_new = torch.tanh(W_xh(x[t]) + W_hh(hidden_states[-1]))
27    h_new.retain_grad()
28    hidden_states.append(h_new)
29
30# Loss = sum of final hidden state (so dL/dh_T = 1)
31loss = hidden_states[-1].sum()
32loss.backward()
33
34# Inspect how the gradient norm shrinks as we go back in time
35for t, h_t in enumerate(hidden_states[1:], 1):
36    print(f"t={t:2d}: ||dL/dh_t|| = {h_t.grad.norm().item():.3e}")

The printed gradient norms tell the whole story: at $t=20$ the gradient norm is 2.00, at $t=1$ it is 2.35 × 10⁻⁴. The first hidden state receives a learning signal 10,000 times weaker than the last. Whatever information about the earliest input is encoded in $h_1$, gradient descent can barely move its weights — the update step is buried in numerical noise.

Long-Term Dependencies in Practice

Why does this mathematical curiosity matter? Because language is full of long-distance dependencies. Consider:

“The cats, which we adopted from the shelter that opened last year, sit quietly.”

The verb must agree with cats (plural), but 13 tokens sit between them. For a vanilla RNN with vanishing gradients, by the time we are predicting “sit”, the signal from “cats” has been multiplied by 13 copies of $\operatorname{diag}(1-h^2)\, W_{hh}$. With a typical gradient decay factor of ~0.7 per step, that is $0.7^{13} \approx 0.01$ — the subject is effectively invisible at the verb position.

Below you can select sentences of varying distance between subject and verb and watch the gradient signal decay as it flows back through the tokens. Short-distance agreement is easy; long-distance agreement is nearly hopeless.

A Historical Detour: Who Discovered This?

The vanishing gradient problem was not understood until almost a decade after backpropagation was popularized. Early RNN practitioners knew their models struggled on long sequences but could not explain why. The first rigorous analysis came from Sepp Hochreiter's 1991 diploma thesis at TU Munich — a largely overlooked document at the time. Yoshua Bengio, Patrice Simard, and Paolo Frasconi gave the problem broader visibility in 1994. Six years later, Hochreiter and Jürgen Schmidhuber would publish the paper that solved it: LSTM.

The Roadmap to LSTM and GRU

If the problem is that every backward step multiplies the gradient, the fix must change that. There are really only three options:

1. Change initialization. Initialize $W_{hh}$ as identity (IRNN), orthogonal, or unitary so its eigenvalues sit on the unit circle. This delays vanishing but does not eliminate it — once the weights start training, the spectral radius drifts.
2. Change the activation path. Use ReLU or LeakyReLU so the derivative is 1 wherever the activation is positive. Helps, but leaves the matrix product problem untouched.
3. Change the architecture. Introduce an additive path alongside the multiplicative one: the cell state in an LSTM, or the gated combination in a GRU. Gradients travel along the additive path almost unchanged, bypassing the vanishing problem entirely. This is what actually worked.

Chapter 17 will build LSTMs and GRUs from these principles, and you will see that the “complicated” gates are just the smallest machinery needed to give gradients a non-multiplicative highway through time. Once you understand vanishing gradients, LSTM stops looking mysterious — it looks inevitable.

The design principle. If you want a neural network to remember something across many steps, you must build a path along which the gradient is not continuously multiplied by a learned matrix. Additive skip connections — whether in LSTMs, GRUs, residual networks, or transformers — are the single most important structural idea in modern deep learning.

Test Your Understanding

Before moving on to LSTM, work through these questions to make sure the mathematics is solid. Each wrong answer comes with an explanation.

Key Takeaways

• The gradient is a product, not a sum. $\partial h_T/\partial h_t$ is a chain of $T-t$ Jacobians multiplied together.
• Vanilla RNNs share weights across time. Every factor in that product involves the same $W_{hh}$ — so its spectral radius dominates.
• Spectral radius < 1 → vanishing; spectral radius > 1 → exploding. The edge of stability ($\rho = 1$) is a knife-edge that training cannot maintain.
• Activation functions shrink gradients further. Sigmoid caps the per-step factor at 0.25; tanh saturates once activations leave $[-1, 1]$.
• Exploding gradients are easy to detect and fix (clip them). Vanishing gradients are silent and devastating — the network simply never learns long-range patterns.
• The architectural fix is an additive gradient path. LSTM, GRU, residual nets, and transformer skip-connections all share this idea: provide a route through the computation where gradients are not multiplied by a learned matrix at every step.
• Hochreiter (1991) and Bengio et al. (1994) gave the problem its rigorous analysis. Hochreiter and Schmidhuber's 1997 LSTM paper gave the first clean solution.