Recurrent Networks & LSTM Cells

Working Memory: a Running Summary

When you read this sentence, you maintain a running summary of the words that came before. By the time you reach the period you have compressed a couple of dozen tokens into a small mental state — enough to disambiguate this, they, that on the next line. Your hippocampus does it for navigation; your auditory cortex does it for music; your frontal cortex does it for plans that span minutes. Recurrent neural networks are the engineering caricature of that capacity.

For RUL prediction the same idea applies to a different kind of memory: as a turbofan's sensor stream rolls past, the model needs to integrate slow degradation patterns over many cycles. A single spike at cycle 23 means little; a slow upward drift over cycles 5-30 means the engine is dying. Convolutions (Section 3.2) only see $K$ cycles at a time. RNNs carry a hidden state across the entire window.

Vanilla RNN and Its Failure Mode

The simplest recurrent cell — the “vanilla RNN” — has one update equation:

$h_t \;=\; \tanh\!\bigl(W_x \, x_t + W_h \, h_{t-1} + b\bigr).$

The hidden state at step $t$ is a learned non-linear function of the current input and the previous hidden state. Repeat for $T$ steps and you have read the whole sequence. Beautiful, simple, and broken on long sequences.

The problem is the gradient. Backpropagating through a length-T RNN produces a product of T Jacobians, each containing a $\tanh'$ derivative bounded above by 1. With $T = 30$ the gradient at the first cycle is attenuated by roughly $0.5^{30} \approx 10^{-9}$ in the worst case. The model effectively cannot learn dependencies more than ~10 cycles long. This is the famous vanishing gradient problem.

The LSTM Cell: Four Gates

An LSTM cell maintains two state vectors that propagate through time: $c_t \in \mathbb{R}^{H}$ — the cell state, a long-term memory; and $h_t \in \mathbb{R}^{H}$ — the hidden output, what downstream layers see. Three gates and one candidate regulate the update:

Together they update the cell and hidden state via

$c_t = f_t \odot c_{t-1} + i_t \odot g_t, \qquad h_t = o_t \odot \tanh(c_t).$

The element-wise $\odot$ is critical: the cell state is updated by simple addition and pointwise scaling — no matrix multiplication in the recurrence loop. That is why gradients survive across long sequences: the partial derivative $\partial c_t / \partial c_{t-1} = f_t$, which is element-wise close to 1 when the forget gate is open.

Interactive: Step Through 8 Timesteps

The trace below runs a scalar LSTM cell on the input pulse $[0, 0, 0, 1, 1, 1, 0, 0]$. Press play or step one timestep at a time. Watch the cell state ramp up when the pulse arrives, hold near 1 while the pulse is active, then decay slowly back toward zero after the pulse ends — the LSTM is remembering the recent input.

The forget gate stays at $f \approx 0.731$ throughout because the input does not affect it (we hard-coded $W_{if} = 0$). 73% of the cell state survives each step, which is what gives the post-pulse decay its characteristic half-life. In a real LSTM the forget gate is learned and can range from 0 (forget everything) to nearly 1 (remember forever).

Interactive: The Unrolled Diagram

Another way to see an RNN is to unroll it across time — draw one cell per timestep, with arrows showing how the hidden state flows from left to right. The animation below unrolls a generic recurrent cell over six timesteps; the same picture applies to any RNN variant including LSTMs.

Python: An LSTM Cell From Scratch

Twenty-five lines of NumPy and the entire algorithm is exposed. We write a scalar LSTM cell (one input dim, one hidden dim) so the gates are visible as actual numbers. Generalising to vector cells is trivial — replace each scalar weight with a matrix, each scalar multiplication with a matmul, and the algebra of the four gates is unchanged.

1import numpy as np
2
3# ----- A scalar LSTM cell, written from scratch -----
4# input_size = hidden_size = 1, so each "matrix" is just a number.
5# Generalises trivially: replace scalars with matmuls and you have nn.LSTMCell.
6class LSTMCellMicro:
7    def __init__(self):
8        self.W_ii, self.b_i = 0.5, 0.0
9        self.W_if, self.b_f = 0.0, 1.0     # forget bias = 1: start "remembering"
10        self.W_ig, self.b_g = 0.8, 0.0
11        self.W_io, self.b_o = 1.0, 0.0
12
13    def step(self, x: float, h_prev: float, c_prev: float):
14        sig = lambda z: 1.0 / (1.0 + np.exp(-z))
15        i = sig(self.W_ii * x + self.b_i)        # input gate in (0, 1)
16        f = sig(self.W_if * x + self.b_f)        # forget gate in (0, 1)
17        g = np.tanh(self.W_ig * x + self.b_g)    # candidate in (-1, 1)
18        o = sig(self.W_io * x + self.b_o)        # output gate in (0, 1)
19        c = f * c_prev + i * g                   # CELL STATE UPDATE
20        h = o * np.tanh(c)                       # HIDDEN OUTPUT
21        return h, c, (i, f, g, o)
22
23
24# ----- Run on a square pulse -----
25cell = LSTMCellMicro()
26seq = [0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0]
27h, c = 0.0, 0.0
28for t, x in enumerate(seq):
29    h, c, (i, f, g, o) = cell.step(x, h, c)
30    print(f"t={t}  x={x:+.0f}  i={i:.3f}  f={f:.3f}  g={g:+.3f}  "
31          f"o={o:.3f}  c={c:+.3f}  h={h:+.3f}")
32
33# t=3  x=+1  i=0.622  f=0.731  g=+0.664  o=0.731  c=+0.413  h=+0.286
34# t=5  x=+1  i=0.622  f=0.731  g=+0.664  o=0.731  c=+0.936  h=+0.536
35# t=7  x=+0  i=0.500  f=0.731  g=+0.000  o=0.500  c=+0.500  h=+0.231

The numbers tell the story

Read down the cell-state column: 0 → 0 → 0 → 0.413 → 0.716 → 0.936 → 0.685 → 0.500. The cell state builds up while the pulse is active and decays gracefully after it ends. That is what working memory looks like in arithmetic.

PyTorch: nn.LSTM in Six Lines

Production code never writes the cell from scratch. PyTorch's nn.LSTM wraps a CUDA-optimised batched implementation with multi-layer and bidirectional support built in.

1import torch
2import torch.nn as nn
3
4# ----- A two-layer bidirectional LSTM, batch-first -----
5torch.manual_seed(0)
6B, T, F = 2, 30, 17                # 2 engines, 30 cycles, 17 sensors
7
8rnn = nn.LSTM(
9    input_size=17,
10    hidden_size=256,
11    num_layers=2,
12    bidirectional=True,
13    batch_first=True,              # CRITICAL: default is False!
14)
15
16# Forward pass on a fake batch
17x = torch.randn(B, T, F)
18out, (h_n, c_n) = rnn(x)
19
20print("input  x   :", tuple(x.shape))     # (2, 30, 17)
21print("output out :", tuple(out.shape))    # (2, 30, 512)
22print("hidden h_n :", tuple(h_n.shape))    # (4, 2, 256)
23print("cell   c_n :", tuple(c_n.shape))    # (4, 2, 256)
24print("# params  :", sum(p.numel() for p in rnn.parameters()))
25# # params  : 2,140,160

Recurrent Networks Beyond RUL

Every row shipped a state-of-the-art result at some point in the last decade. Transformers (Section 3.4) have since taken over large-scale text and vision, but LSTMs remain the right tool for small-batch, low-latency, low-data settings — including most prognostic problems with under a million sensor samples.

The Three Pitfalls

The point. An LSTM is a learnable, gated working memory. The gates are the engineering trick that solves vanishing gradients; the cell state is the long-term memory; the hidden output is what the rest of the model sees.

Takeaway

• RNNs read sequences and update a hidden state. They add temporal modelling on top of whatever frontend you put in place (Conv1D in this book).
• Vanilla RNNs vanish. Length-30 sequences are too long for tanh-based recurrences; gradients shrink to nothing.
• LSTMs fix it with three gates and a candidate. $c_t = f_t \odot c_{t-1} + i_t \odot g_t$ and $h_t = o_t \odot \tanh(c_t)$. Gradients flow through $c_t$ almost unattenuated.
• PyTorch's nn.LSTM is six arguments. input_size, hidden_size, num_layers, bidirectional, batch_first, dropout — the entire backbone of Chapter 9 fits on one line.
• Always set batch_first=True. The default is $(T, B, F)$ and breaks every other layer's shape contract.