LSTM and GRU Formulations

Learning Objectives

By the end of this section, you will:

1. Understand the gating mechanism as the key innovation that enables long-range learning
2. Derive all four LSTM gates and understand each gate's role in controlling information flow
3. Trace the cell state as the highway for gradient propagation
4. Prove how LSTMs prevent vanishing gradients through additive updates
5. Compare GRU formulations and understand when to choose each architecture
6. Understand bidirectional processing and why we use it for RUL prediction

Why This Matters: Our AMNL model uses a BiLSTM (bidirectional LSTM) as its core sequence processor. Understanding how gates control information flow explains why our model can capture both short-term spikes and long-term degradation trends—essential for accurate RUL prediction across all operating conditions.

The Gating Intuition

Before diving into equations, let's build intuition. The vanilla RNN had a fundamental problem: the same transformation was applied at every timestep, with no ability to control what information to keep or discard.

The Gate Metaphor

Think of a gate as a learned, differentiable switch that outputs values between 0 and 1:

• Gate = 0: Block everything (the gate is closed)
• Gate = 1: Pass everything (the gate is open)
• Gate = 0.7: Allow 70% through (partially open)

Mathematically, if $\mathbf{g} \in [0, 1]^d$ is a gate vector and $\mathbf{x} \in \mathbb{R}^d$ is some signal, then:

Where $\odot$ is element-wise multiplication. Each dimension can be gated independently!

Why Sigmoid Creates Gates

Gates use the sigmoid activation $\sigma(z) = \frac{1}{1 + e^{-z}}$ because:

1. Output range: Always in $(0, 1)$
2. Smooth and differentiable: Allows gradient-based learning
3. Saturation: For extreme inputs, output approaches 0 or 1

LSTM Architecture

The Long Short-Term Memory (LSTM) cell, introduced by Hochreiter & Schmidhuber (1997), addresses vanilla RNN limitations with two key innovations:

Innovation 1: The Cell State

LSTMs introduce a separate cell state $\mathbf{C}_t$ that runs through the entire sequence with only linear interactions:

This is an additive update (not multiplicative like vanilla RNNs). Information can flow unchanged for many timesteps—the gradient highway!

Innovation 2: Three Gates

LSTMs use three gates to control information flow:

Information Flow Overview

At each timestep, the LSTM:

1. Forget: Decide what to remove from cell state
2. Store: Create new candidate information
3. Update: Combine old (filtered) and new information
4. Output: Produce hidden state from cell state

LSTM Equations Derived

Let's derive each LSTM component systematically. Our AMNL model uses hidden dimension $H = 64$.

Notation

Step 1: Concatenate Input

All gates share the same input vector:

Where $[\cdot; \cdot]$ denotes concatenation. This combines history (hidden state) with new input.

Step 2: Forget Gate

Dimension: $\mathbf{f}_t \in (0, 1)^H$

Intuition: For each dimension of the cell state, how much should we retain from the previous timestep?

Step 3: Input Gate and Candidate State

The input gate controls how much new information to add:

The candidate cell state is what new information to add:

Key insight: Separating "how much" (gate) from "what" (candidate) allows independent control. The candidate uses tanh to produce values in $(-1, 1)$—it can push the cell state in either direction.

Step 4: Cell State Update

This is the critical equation! Breaking it down:

• $\mathbf{f}_t \odot \mathbf{C}_{t-1}$: Selectively forget old cell state
• $\mathbf{i}_t \odot \tilde{\mathbf{C}}_t$: Selectively add new information
• Addition (+): Not multiplication! Gradients flow directly.

Step 5: Output Gate and Hidden State

Intuition: The cell state $\mathbf{C}_t$ contains all stored information, but we may not want to expose all of it. The output gate filters what becomes visible in the hidden state.

Complete LSTM Equations

For reference, here are all equations together:

How LSTM Solves Vanishing Gradients

Now we can prove why LSTMs don't suffer from vanishing gradients. Consider the gradient of the loss with respect to an earlier cell state.

Cell State Gradient

From the cell update equation $\mathbf{C}_t = \mathbf{f}_t \odot \mathbf{C}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{C}}_t$:

The gradient from time $t$ to time $k$ along the cell state path:

Comparison with Vanilla RNN

Why This Prevents Vanishing

The Constant Error Carousel

When $\mathbf{f}_t = \mathbf{1}$ and $\mathbf{i}_t = \mathbf{0}$:

The cell state is copied exactly! This "constant error carousel" allows information and gradients to persist indefinitely—solving the core problem of vanilla RNNs.

GRU: A Streamlined Alternative

The Gated Recurrent Unit (GRU), introduced by Cho et al. (2014), simplifies the LSTM by merging gates and eliminating the separate cell state.

GRU Design Philosophy

GRU asks: "Can we get LSTM-like benefits with fewer parameters?" The answer is yes, through clever gate merging:

GRU Equations

Reset Gate: How much of the past to forget when computing the candidate:

Update Gate: How much of the new candidate to use versus keeping old state:

Candidate Hidden State: Proposed new state (reset gate filters history):

Hidden State Update: Interpolation between old and new:

Key Insight: Complementary Gating

The GRU update uses $(1 - \mathbf{z}_t)$ and $\mathbf{z}_t$ as complementary weights:

• $\mathbf{z}_t \approx 0$: Keep old state, ignore new input
• $\mathbf{z}_t \approx 1$: Replace with new candidate
• $\mathbf{z}_t = 0.5$: Equal mix of old and new

This is analogous to LSTM's forget and input gates, but with a constraint: $\mathbf{f}_t + \mathbf{i}_t = 1$ (implicitly).

LSTM vs GRU Comparison

Parameter Count

Empirical Comparison

Why We Use LSTM for RUL

For our AMNL model, we chose LSTM because:

1. Long-range dependencies: Degradation patterns may span the entire 30-timestep window
2. Separate cell state: Better for preserving subtle trend information across timesteps
3. Dataset size: C-MAPSS has sufficient data to train LSTM without overfitting
4. Proven performance: LSTM is well-established for industrial time series

Design Choice: For RUL prediction, the additional capacity of LSTM's separate cell state outweighs GRU's training efficiency. When long-term memory matters most, LSTM is the safer choice.

Bidirectional Processing

Standard RNNs/LSTMs process sequences left-to-right, computing $\overrightarrow{\mathbf{h}}_t$ from past context. But what about future context?

The Bidirectional Idea

A Bidirectional LSTM (BiLSTM) runs two LSTMs:

1. Forward LSTM: Processes $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_T$ left-to-right
2. Backward LSTM: Processes $\mathbf{x}_T, \mathbf{x}_{T-1}, \ldots, \mathbf{x}_1$ right-to-left

BiLSTM Equations

Forward pass:

Backward pass:

Combined output:

The output at each timestep concatenates forward and backward hidden states, giving dimension $2H = 128$ in our model.

Why BiLSTM for RUL?

Consider a sensor reading at timestep 15 in a 30-step window:

• Forward context: What happened before (cycles 1-14)
• Backward context: What happens after (cycles 16-30)

For RUL prediction, both matter! A temperature spike at cycle 15 is interpreted differently if:

• It returns to normal (backward context shows recovery)
• It continues rising (backward context shows degradation)

Parallel Computation

An important practical note: forward and backward passes are independent and can run in parallel! This makes BiLSTM only ~1.5× slower than unidirectional LSTM, not 2×.

Summary

In this section, we derived the mathematics of LSTM and GRU architectures:

1. Gates are learned sigmoid functions that control information flow with values in $(0, 1)$
2. LSTM has four components: forget gate ($\mathbf{f}_t$), input gate ($\mathbf{i}_t$), candidate ($\tilde{\mathbf{C}}_t$), output gate ($\mathbf{o}_t$)
3. Cell state update uses addition: $\mathbf{C}_t = \mathbf{f}_t \odot \mathbf{C}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{C}}_t$
4. Gradient flow through cell state is $\prod_i \mathbf{f}_i$—controlled by learned gates, not fixed weights
5. GRU simplifies to 3 gates with complementary gating: $\mathbf{h}_t = (1 - \mathbf{z}_t) \odot \mathbf{h}_{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t$
6. BiLSTM captures both past and future context by running forward and backward LSTMs

Looking Ahead: LSTMs process sequences step-by-step, compressing all history into a fixed-size vector. But what if we want to directly access any past timestep? The attention mechanism solves this by computing a weighted combination of all hidden states, allowing the model to focus on the most relevant parts of the sequence. This is especially powerful for RUL prediction where specific degradation events matter more than others.

With LSTM and BiLSTM understood, we are ready to explore how attention mechanisms further enhance sequence modeling.