GRU Architecture

Why the GRU Exists

The LSTM was a breakthrough — finally, a recurrent cell that could remember across many timesteps. But by 2014, researchers had noticed something curious: the LSTM has four sub-networks (forget, input, candidate, output) operating on two state vectors ($h_t$ and $C_t$). Was all that machinery really necessary? Or could a simpler design solve the same vanishing-gradient problem with fewer moving parts?

In Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation, Kyunghyun Cho and colleagues proposed the Gated Recurrent Unit. It makes two ruthless simplifications. First, the cell state and hidden state are merged into a single vector $h_t$. Second, the forget and input gates are coupled into a single update gate whose fractions sum to 1 by construction. A separate reset gate remains, but it plays a very narrow role — it only affects the candidate, not the memory highway.

The core bet: we don't need two independent gates for forgetting and writing. If we keep $1-z$ of the old memory, we have exactly $z$ budget left for new content. One knob, two effects, no loss of expressivity on most real tasks.

The result is a cell with three weight matrices instead of four, a single state vector, and — empirically — performance that tracks the LSTM closely on language modelling, machine translation, and music generation while training faster. For many practitioners the GRU is the default; for others the LSTM's extra flexibility earns its keep. Understanding the GRU's minimalism is the best way to appreciate both.

One State Vector, Two Gates

An LSTM carries two vectors through time: the cell state $C_t$ (internal memory) and the hidden state $h_t$ (external output). A GRU carries only $h_t$. That single vector plays both roles — it is the memory that persists and the signal that flows to the next layer. This is not just notation: it is a genuine reduction in the cell's internal degrees of freedom.

The three learned linear layers are:

And the single update rule that glues them together:

$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$

The Update Gate: One Knob Replaces Two

The update gate $z_t \in (0, 1)^d$ is a per-channel sigmoid. Its value at channel $c$ answers a single question: what fraction of the new candidate do I write into channel $c$ of memory? The remaining fraction $1 - z_c$ automatically goes to keeping the old value. Three extreme settings show the gate's range:

Compare this to the LSTM, where the forget gate $f_t$ and input gate $i_t$ are two independent sigmoids. An LSTM can set $f = 0.9, i = 0.9$ — simultaneously keeping a lot of the old memory AND writing a lot of new content (letting the cell state grow in magnitude). A GRU cannot express that combination. If it keeps 90% of the past, it writes at most 10% of the candidate.

The Reset Gate: Forgetting Just for the Proposal

The reset gate is the more subtle of the two gates, and the one most beginners misunderstand. Its equation looks almost identical to the update gate's:

$r_t = \sigma(W_r \cdot [h_{t-1}, x_t] + b_r)$

But its role in the network is completely different. The reset gate does not appear in the final mixing step. It appears only inside the candidate:

$\tilde{h}_t = \tanh(W_h \cdot [r_t \odot h_{t-1}, \, x_t] + b_h)$

Look carefully at what $r_t$ multiplies: $h_{t-1}$, but only for the purpose of building the proposal. When $r_c \approx 0$ on some channel, that channel of the past is hidden from the candidate — the proposal depends essentially on $x_t$ alone for that channel. When $r_c \approx 1$, the past participates fully.

The Candidate Hidden State

With the reset gate in hand we can build $\tilde{h}_t$:

$\tilde{h}_t = \tanh(W_h \cdot [r_t \odot h_{t-1}, \, x_t] + b_h)$

Three things are worth pausing on. First, the activation is $\tanh$, not $\sigma$. Gates are soft switches in $(0, 1)$ — they should never flip signs. The candidate is content, and content must be able to cancel or flip previous content, so it lives in $(-1, 1)$.

Second, the candidate's input is $[r_t \odot h_{t-1}, \, x_t]$ — reset-masked past concatenated with the current input. The update gate $z_t$ saw raw $h_{t-1}$; the candidate sees a possibly-silenced version. This asymmetry is a deliberate design choice that makes the reset gate's role well-defined: shape the proposal, not the keep-fraction.

Third, $\tilde{h}_t$ is just a proposal. It does not become the new hidden state on its own. The update gate in the next step decides how much of it to actually commit.

The Final Mix: Convex Combination in Time

Every piece has been assembled; the final step is mechanical:

$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$

Per channel this is a convex combination — a weighted average whose weights sum to 1. Geometrically, $h_t$ lies on the line segment between the old memory $h_{t-1}$ and the new candidate $\tilde{h}_t$, at the position dictated by $z$.

Why is this equation the heart of the GRU? Because of what it says about gradients. Taking a derivative of $h_t$ with respect to $h_{t-1}$ (treating gates as constants for the moment):

$\frac{\partial h_t}{\partial h_{t-1}} \approx (1 - z_t) \cdot I + z_t \cdot \frac{\partial \tilde{h}_t}{\partial h_{t-1}}$

The first term is a scaled identity matrix. When $z_t \approx 0$ (keep memory), the gradient flows backward essentially unchanged — no repeated multiplication by small numbers, no vanishing. This is exactly the same mechanism that makes the LSTM's cell state a gradient highway, accomplished with one fewer gate and one fewer state vector.

The unifying picture: LSTM and GRU both solve the vanishing-gradient problem by inserting an additive identity path into the recurrence. The LSTM's path runs on $C$; the GRU's runs on $h$ itself. Once you see that, the surface differences (number of gates, number of states) become a matter of taste and parameter count.

Interactive: GRU Architecture

Click through the four stages — reset gate, candidate, update gate, final mix — and watch each block of the cell light up. The dashed orange line is the identity path $(1 - z_t) \odot h_{t-1}$ that lets gradients travel through time unchanged.

Interactive: The Gates at Work

The explorer uses scalar versions of $r_t$, $z_t$, and $\tilde{h}_t$ so you can watch all three values respond to the two sliders in real time. In a full network each gate is a vector and each channel acts independently — but the qualitative behavior is exactly what the scalar case shows.

Interactive: GRU vs LSTM Side by Side

The following visualization puts the two architectures next to each other. You can slide the input and hidden dimensions and watch the parameter count update — the GRU consistently uses $3/4$ of the LSTM's parameters because it has three gate/candidate matrices instead of four, and no separate cell state.

A Full Forward Pass with Real Numbers

Before we look at code, let us trace one GRU cell by hand on the three-token toy sequence “I love it”. We use the same two-dimensional toy embeddings as the LSTM section — this makes every matmul a 2×4 matrix times a (4,) vector, small enough to write out.

Weights

Step 1 — token “I”, x = [0.5, −0.2]

Concatenate: $[h_0, x_1] = [0, 0, 0.5, -0.2]$. Compute the reset pre-activation row by row:

• Row 0 of $W_r [h, x]$: $0.3 \cdot 0 + (-0.2)\cdot 0 + 0.4\cdot 0.5 + 0.1\cdot(-0.2) = 0.18$; add bias 0.1 → $0.28$.
• Row 1: $0.1\cdot 0 + 0.5\cdot 0 + (-0.3)\cdot 0.5 + 0.2\cdot(-0.2) = -0.19$; add bias 0.0 → $-0.19$.
• $r_1 = \sigma([0.28, -0.19]) = [0.5695, 0.4526]$.

Update gate, same drill but with $W_z, b_z$: $z_1 = \sigma([-0.23, 0.31]) = [0.4428, 0.5769]$.

Candidate: because $h_0 = 0$, the reset-masked past is $r \odot h_0 = 0$, so $[r \odot h_0, x_1] = [0, 0, 0.5, -0.2]$.

• Row 0 of $W_h$: $0.3\cdot 0.5 + 0.2\cdot(-0.2) + 0.0 = 0.11$.
• Row 1: $(-0.1)\cdot 0.5 + 0.5\cdot(-0.2) + 0.1 = -0.05$.
• $\tilde{h}_1 = \tanh([0.11, -0.05]) = [0.1096, -0.0500]$.

Final mix: $h_1 = (1 - z_1) \odot 0 + z_1 \odot \tilde{h}_1 = [0.4428\cdot 0.1096,\; 0.5769\cdot(-0.05)] = [0.0485, -0.0288]$.

Step 2 — token “love”, x = [0.8, 0.3]

Now the reset gate has real memory to act on. After the arithmetic (exactly the same recipe, this time with $h_1 = [0.0485, -0.0288]$):

Step 3 — token “it”, x = [0.1, 0.9]

Compare this to the LSTM section's worked example. There, the cell state grew in magnitude from step 1 to step 3 (≈0.21 → 0.35) while the hidden state stayed small. Here, the single$h$ vector grows from ≈0.05 to ≈0.35 across three steps — the GRU does not split memory from output, so you see the accumulation directly in the exposed vector.

GRU From Scratch in Python

Every line of the forward pass you just traced by hand shows up as a line of NumPy below. Click any line in the code — the explanation panel reveals shapes, values, and the mathematical role of every symbol at that point in the computation. This is the same cell you just computed; running it will reproduce the hand-traced numbers exactly.

    h0     h1     x0     x1
c0  0.3  -0.2   0.4    0.1
c1  0.1   0.5  -0.3    0.2

    h0     h1     x0     x1
c0  0.2   0.3  -0.1    0.4
c1 -0.2   0.1   0.5    0.2

     rh0    rh1    x0     x1
c0  0.1  -0.4   0.3    0.2
c1  0.4   0.2  -0.1    0.5

1import numpy as np
2
3def sigmoid(x):
4    return 1.0 / (1.0 + np.exp(-x))
5
6# ---- One GRU cell: processes a single timestep ----
7def gru_cell(x_t, h_prev, params):
8    # Concatenate previous hidden state with current input: shape (hidden + input,)
9    z_in = np.concatenate([h_prev, x_t])
10
11    # Two gates — reset r and update z, each a sigmoid of a linear layer
12    r_t = sigmoid(params["W_r"] @ z_in + params["b_r"])   # reset gate
13    z_t = sigmoid(params["W_z"] @ z_in + params["b_z"])   # update gate
14
15    # Candidate hidden state — reset gate acts ONLY here, on h_prev
16    rh = r_t * h_prev
17    cand_in = np.concatenate([rh, x_t])
18    h_tilde = np.tanh(params["W_h"] @ cand_in + params["b_h"])
19
20    # Final hidden state — convex combination (1-z) keeps old, z writes new
21    h_t = (1.0 - z_t) * h_prev + z_t * h_tilde
22
23    return h_t
24
25# ---- Run over a sequence ----
26params = {
27    "W_r": np.array([[ 0.3, -0.2,  0.4,  0.1],
28                     [ 0.1,  0.5, -0.3,  0.2]]),
29    "b_r": np.array([ 0.1,  0.0]),
30    "W_z": np.array([[ 0.2,  0.3, -0.1,  0.4],
31                     [-0.2,  0.1,  0.5,  0.2]]),
32    "b_z": np.array([-0.1,  0.1]),
33    "W_h": np.array([[ 0.1, -0.4,  0.3,  0.2],
34                     [ 0.4,  0.2, -0.1,  0.5]]),
35    "b_h": np.array([ 0.0,  0.1]),
36}
37
38sequence = [np.array([0.5, -0.2]),   # "I"
39            np.array([0.8,  0.3]),   # "love"
40            np.array([0.1,  0.9])]   # "it"
41
42h = np.zeros(2)
43for t, x_t in enumerate(sequence, 1):
44    h = gru_cell(x_t, h, params)
45    print(f"step {t}: h={h.round(4)}")

Running the code prints:

Notice that there is no separate $C_t$ column. The GRU only has one state, and what you see is what downstream layers receive.

The Same GRU in PyTorch

PyTorch ships two ways to use a GRU: nn.GRUCell, which mirrors our NumPy loop exactly (one step at a time, you manage the recurrence in Python), and nn.GRU, which handles the whole sequence in a single optimized call. Both use the same underlying arithmetic; they differ in where the time-loop lives.

1import torch
2import torch.nn as nn
3
4# ---- nn.GRUCell: one step at a time (mirrors our NumPy loop) ----
5torch.manual_seed(0)
6cell = nn.GRUCell(input_size=2, hidden_size=2)
7
8sequence = torch.tensor([[0.5, -0.2],
9                         [0.8,  0.3],
10                         [0.1,  0.9]])
11
12h = torch.zeros(1, 2)                 # (batch=1, hidden=2)
13for t, x_t in enumerate(sequence, 1):
14    x_t = x_t.unsqueeze(0)            # (1, 2) batch dim
15    h = cell(x_t, h)
16    print(f"step {t}: h={h.detach().numpy().round(4)}")
17
18# ---- nn.GRU: whole sequence in one call ----
19gru = nn.GRU(input_size=2, hidden_size=2, batch_first=True)
20batch = torch.tensor([[[0.5, -0.2], [0.8, 0.3], [0.1, 0.9]]])  # (1, 3, 2)
21output, h_n = gru(batch)
22print("output shape:", tuple(output.shape))
23print("final h_n   :", h_n.detach().numpy().round(4))

The printed numbers will differ from the NumPy version because PyTorch uses random weight initialization instead of our hand-picked matrices. The mechanism is identical, line for line — three linear projections, two sigmoid gates, one tanh candidate, one convex mix.

Why GRUs Still Solve Vanishing Gradients

The canonical objection to vanilla RNNs is that gradients vanish when backpropagated through many timesteps because every step multiplies by the same small Jacobian. The LSTM escapes this via an additive cell state whose recurrence has an identity term. Does the GRU — with its single state — really preserve the same property?

Yes. Differentiate the update rule:

$\frac{\partial h_t}{\partial h_{t-1}} = \text{diag}(1 - z_t) \;+\; \text{diag}(z_t) \cdot \frac{\partial \tilde{h}_t}{\partial h_{t-1}} \;+\; (\text{terms from } \partial z_t / \partial h_{t-1})$

The first term, $\text{diag}(1 - z_t)$, is a diagonal matrix whose entries are the keep-fractions. When the update gate chooses to preserve channel $c$, that entry is close to 1, so the gradient of $h_{t,c}$ with respect to $h_{t-1,c}$ is close to 1 — no shrinkage. Over $T$ steps the backward path is a product of such near-identity factors, which stays bounded.

The additional terms come from gradients flowing through the gates and the candidate tanh. These paths do decay — but they are not the only path back. The identity path ensures that a non-vanishing gradient always exists, and that is enough to let learned weights find long-range dependencies.

Interactive: Gradient Flow (RNN vs LSTM vs GRU)

The three curves below are the same $\|\partial \ell_t / \partial \text{state}_1\|$ measurement run through each cell type on a $\log_{10}$ axis. Slide the keep-fraction control — it is $f_t$ for the LSTM and $1 - z_t$ for the GRU. Both gated cells stay near 1; only the vanilla RNN collapses.

Numerical Gradient Check

To make the argument concrete, compare the highway approximation $(1 - z_3) \odot (1 - z_2)$ to the numerical Jacobian of the three-step GRU from §17.2's worked example. We use finite differences (step $\varepsilon = 10^{-6}$) — autograd would give the same answer, exactly, for free.

    h0     h1     x0     x1
c0  0.3  -0.2   0.4    0.1
c1  0.1   0.5  -0.3    0.2

    h0     h1     x0     x1
c0  0.2   0.3  -0.1    0.4
c1 -0.2   0.1   0.5    0.2

     rh0    rh1    x0     x1
c0  0.1  -0.4   0.3    0.2
c1  0.4   0.2  -0.1    0.5

1import numpy as np
2
3# Update-gate values from §17.2's worked example, copied over.
4z_2 = np.array([0.4853, 0.6335])   # at t=2
5z_3 = np.array([0.5780, 0.5760])   # at t=3
6
7# ---- Analytical highway term ----
8# Just as the LSTM has ∂C_t/∂C_{t-1} = f_t, the GRU has
9#     ∂h_t / ∂h_{t-1}  ≈  diag(1 − z_t)   (ignoring the small tanh-candidate path).
10# So the highway contribution to ∂h_3/∂h_1 is (1 − z_3) ⊙ (1 − z_2):
11dh3_dh1_highway = (1 - z_3) * (1 - z_2)
12print(f"(1 - z_2) = {(1 - z_2).round(4)}")
13print(f"(1 - z_3) = {(1 - z_3).round(4)}")
14print(f"highway   ∂h_3/∂h_1 (diag) ≈ {dh3_dh1_highway.round(4)}")
15
16# ---- Finite-difference verification ----
17# Rerun the full GRU forward pass from §17.2 (same weights) but perturb
18# each channel of h_0 by ε and observe the effect on h_3.  The ratio
19# (Δh_3 / Δh_0) is the numerical Jacobian diagonal — agreeing with autograd
20# to O(ε) and with the highway term to leading order.
21def sigmoid(x):
22    return 1.0 / (1.0 + np.exp(-x))
23
24W_r = np.array([[ 0.3, -0.2,  0.4,  0.1],
25                [ 0.1,  0.5, -0.3,  0.2]])
26b_r = np.array([ 0.1,  0.0])
27W_z = np.array([[ 0.2,  0.3, -0.1,  0.4],
28                [-0.2,  0.1,  0.5,  0.2]])
29b_z = np.array([-0.1,  0.1])
30W_h = np.array([[ 0.1, -0.4,  0.3,  0.2],
31                [ 0.4,  0.2, -0.1,  0.5]])
32b_h = np.array([ 0.0,  0.1])
33
34def gru_run(h0):
35    seq = [np.array([0.5, -0.2]),
36           np.array([0.8,  0.3]),
37           np.array([0.1,  0.9])]
38    h = h0.copy()
39    for x in seq:
40        z_in    = np.concatenate([h, x])
41        r       = sigmoid(W_r @ z_in + b_r)
42        z       = sigmoid(W_z @ z_in + b_z)
43        cand_in = np.concatenate([r * h, x])
44        h_tilde = np.tanh(W_h @ cand_in + b_h)
45        h       = (1 - z) * h + z * h_tilde
46    return h
47
48eps   = 1e-6
49h0    = np.zeros(2)
50h3    = gru_run(h0)
51jacob = np.zeros((2, 2))
52for j in range(2):                          # perturb channel j
53    h0_plus      = h0.copy(); h0_plus[j]  += eps
54    h3_plus      = gru_run(h0_plus)
55    jacob[:, j]  = (h3_plus - h3) / eps     # numerical ∂h_3/∂h_0[:, j]
56
57print(f"finite-diff ∂h_3/∂h_0 =\n{jacob.round(4)}")
58print(f"diag of FD Jacobian   = {np.diag(jacob).round(4)}")
59print(f"highway approximation = {dh3_dh1_highway.round(4)}")

The diagonal of the finite-difference Jacobian is $[0.2283, 0.1642]$; the highway term approximation is $[0.2172, 0.1554]$. The ~5% gap is the reset-gate and candidate-tanh contribution autograd would also include. The leading story — an additive identity path that keeps gradients from vanishing — is the highway term.

When to Pick GRU Over LSTM

The empirical literature (Chung et al., 2014; Jozefowicz et al., 2015; Greff et al., 2017) has not produced a clean winner. On many language, speech, and music tasks the two perform within noise of each other. Greff and colleagues ran the definitive ablation — eight LSTM variants (with and without the forget gate, peephole connections, coupled input-forget gate, and others) on 5200 runs covering speech, handwriting, and polyphonic music — and found that none of the mutations beat the vanilla LSTM by a meaningful margin, and the most impactful single change was the forget-bias initialization trick from §17.1. The choice between GRU and LSTM usually rests on three practical factors instead.

Parameter budget and training speed

With hidden size $n$ and input size $m$:

• GRU: $3 \cdot (n \cdot (n + m) + n)$ parameters.
• LSTM: $4 \cdot (n \cdot (n + m) + n)$ parameters.

A GRU uses exactly $75\%$ of the LSTM's parameters for the same hidden size. On a constrained memory or latency budget — edge devices, mobile, real-time ASR — GRU lets you either run faster or spend the savings on a larger hidden state.

Dataset size

With fewer parameters to fit, a GRU tends to generalize slightly better on small datasets and overfit more gracefully. On very large datasets (hundreds of millions of tokens) the LSTM's extra flexibility sometimes pays off, especially at larger hidden sizes.

Task structure

1. Very long sequences with sparse dependencies (e.g., long-document summarization): an LSTM's decoupled forget + input gates sometimes model “remember this AND record that” more precisely.
2. Moderate sequences, streaming inference (speech, real-time translation): GRUs win on latency and usually match accuracy.
3. Encoder in an encoder-decoder: GRU is the modern default in many seq2seq implementations before the rise of Transformers.

Check Your Understanding

Eight questions covering the two gates, the identity path, parameter count, and the practical trade-offs against the LSTM. Each explanation ties the answer back to the equation it came from.

Summary

1. One state, two gates. The GRU carries a single vector $h_t$ through time and uses two gates — reset $r_t$ and update $z_t$ — to control it.
2. The update gate couples forget and input. The rule $h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$ forces keep and write to sum to 1 per channel. One knob, no slack.
3. The reset gate only shapes the candidate. Inside $\tilde{h}_t = \tanh(W_h \cdot [r_t \odot h_{t-1}, x_t] + b_h)$, the reset gate decides how much past context the proposal sees. It never touches the identity path.
4. The identity path preserves gradients. $\partial h_t/\partial h_{t-1}$ contains a $\text{diag}(1 - z_t)$ term — a scaled identity that carries gradients through time when the gate chooses to keep memory.
5. Three matrices, not four. A GRU uses $W_r, W_z, W_h$ — 75% of the LSTM's parameter count for the same hidden size. No cell state means no output gate.
6. Empirically on par. GRU and LSTM perform comparably on most benchmarks. GRU wins on training speed and small datasets; LSTM sometimes wins at very large scales.
7. Same mathematical trick as the LSTM. Both solve vanishing gradients with an additive recurrence — the GRU puts the addition directly on $h$; the LSTM puts it on $C$. The rest is a matter of how aggressively you simplify.

The core idea in one line: $h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$ — a soft slider between keeping memory and writing new content, and that slider is all the GRU needs.

With both cells understood, the next section — §17.3 Implementing Sequence Models in PyTorch — turns the equations into production-grade code. You will build an LSTM from scratch in NumPy, rebuild it in PyTorch, verify the two agree to machine precision, and wire up the full embedding → packed BiLSTM → linear sentiment classifier that was the 2016 workhorse of applied NLP.

References

• Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H. & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. EMNLP 2014 / arXiv:1406.1078. [Introduces the GRU.]
• Chung, J., Gulcehre, C., Cho, K. & Bengio, Y. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. NIPS 2014 Deep Learning Workshop / arXiv:1412.3555. [Compares GRU and LSTM across several sequence tasks; found them roughly comparable, with GRU slightly preferred on polyphonic music.]
• Jozefowicz, R., Zaremba, W. & Sutskever, I. (2015). An Empirical Exploration of Recurrent Network Architectures. ICML 2015. [Large-scale search over gated cell variants; no mutation consistently beat both GRU and LSTM.]
• Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R. & Schmidhuber, J. (2017). LSTM: A Search Space Odyssey. IEEE TNNLS 28(10), 2222–2232 / arXiv:1503.04069. [Eight LSTM variants, none dominating the vanilla architecture; the Coupled Input-Forget Gate variant is effectively the GRU's 1-z/z idea grafted onto an LSTM.]
• Hochreiter, S. & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation 9(8), 1735–1780. [Original LSTM paper — provides the vanishing-gradient analysis that both LSTMs and GRUs address.]