Two-Layer BiLSTM Design

Learning Objectives

By the end of this section, you will:

1. Understand why we stack two BiLSTM layers instead of one
2. Trace data flow through the two-layer architecture
3. Explain hierarchical temporal feature learning
4. Calculate the parameter count for the BiLSTM encoder
5. Apply dropout between layers for regularization

Why This Matters: Stacking LSTM layers creates a hierarchy of temporal abstractions—the first layer learns basic patterns, the second layer learns patterns of patterns. This mirrors the CNN hierarchy and provides richer representations for RUL prediction.

Why Two Layers?

A single BiLSTM layer already captures temporal dependencies. Why add a second?

Analogy: CNN Depth

Just as stacking CNN layers builds increasingly abstract spatial features, stacking LSTM layers builds increasingly abstract temporal features:

Expressiveness Gains

A second layer provides:

• Compositional learning: Layer 2 can learn functions of Layer 1's hidden states
• Non-linear transformations: More depth = more complex mappings
• Hierarchical abstraction: Higher layers see "summaries" of lower layers

Diminishing Returns

Why not three or more layers?

For C-MAPSS with ~20K training windows, two layers provide the best capacity-data balance.

Interactive: LSTM Cell Explorer

Before understanding how we stack BiLSTM layers, let's visualize how a single LSTM cell processes data. The visualizer below uses actual CNN output (8 features × 6 timesteps) and shows the step-by-step gate computations.

Try This: Click through different gates (Forget, Input, Candidate, etc.) to see exactly how LSTM processes information. Notice how the forget gate decides what to keep from the previous cell state, while the input gate controls what new information to add.

Two-Layer Architecture

Our BiLSTM encoder stacks two bidirectional LSTM layers, each with hidden size 128.

Data Flow

1Input from CNN: (batch, 30, 64)
2         ↓
3┌────────────────────────────────────────────────────┐
4│                  BiLSTM Layer 1                    │
5│  Forward:  LSTM(64 → 128)  →  H→₁ ∈ (B, 30, 128)  │
6│  Backward: LSTM(64 → 128)  →  H←₁ ∈ (B, 30, 128)  │
7│  Concat:   H₁ = [H→₁; H←₁] ∈ (B, 30, 256)         │
8└────────────────────────────────────────────────────┘
9         ↓
10    Dropout(p=0.3)
11         ↓
12┌────────────────────────────────────────────────────┐
13│                  BiLSTM Layer 2                    │
14│  Forward:  LSTM(256 → 128) →  H→₂ ∈ (B, 30, 128)  │
15│  Backward: LSTM(256 → 128) →  H←₂ ∈ (B, 30, 128)  │
16│  Concat:   H₂ = [H→₂; H←₂] ∈ (B, 30, 256)         │
17└────────────────────────────────────────────────────┘
18         ↓
19Output: (batch, 30, 256)

Dimension Summary

Interactive: BiLSTM Explorer

Now let's see how bidirectional processing works. The BiLSTM runs two separate LSTM networks—one processing left-to-right (forward), the other right-to-left (backward)—and concatenates their outputs.

Key Insight: At each timestep, the forward LSTM captures "what came before" while the backward LSTM captures "what comes after." The concatenated output gives each position full context from both directions—crucial for understanding degradation patterns in sensor data.

Hierarchical Temporal Features

Each layer learns different levels of temporal abstraction.

Layer 1: Low-Level Temporal Patterns

The first layer directly processes CNN features, learning:

• Immediate transitions: How features change from one timestep to the next
• Short-term memory: Recent history relevant to current state
• Local temporal context: Patterns spanning a few timesteps

Layer 2: High-Level Temporal Patterns

The second layer processes Layer 1's hidden states, learning:

• Patterns of patterns: How low-level dynamics combine
• Long-term trends: Overall trajectory of degradation
• Abstract dynamics: "Accelerating degradation" vs "steady decline"

Parameter Count Analysis

Let us calculate the number of parameters in each BiLSTM layer.

LSTM Parameter Formula

For an LSTM with input size $D$ and hidden size $H$:

The factor of 4 accounts for the four gates (forget, input, candidate, output). Each gate has:

• Weight matrix: $(D + H) \times H$
• Bias: $H$

BiLSTM Layer 1

BiLSTM Layer 2

Total BiLSTM Parameters

The two-layer BiLSTM accounts for approximately 592K of the model's total 3.5M parameters.

Dropout Between Layers

We apply dropout between the two BiLSTM layers to prevent overfitting.

Placement

1Layer 1 output: (B, 30, 256)
2       ↓
3   Dropout(p=0.3)  ← Applied here
4       ↓
5Layer 2 input: (B, 30, 256)

Why Between Layers?

• Prevents co-adaptation: Layer 2 cannot rely on specific Layer 1 neurons
• Regularizes hierarchy: Each layer must be independently useful
• Standard practice: PyTorch's nn.LSTM has a built-in dropout parameter for this

Rate Selection

Summary

In this section, we designed the two-layer BiLSTM encoder:

1. Two layers: Hierarchical temporal abstraction without overfitting
2. Hidden size 128: 256-dim output per layer (bidirectional)
3. Hierarchical features: Layer 1 captures low-level dynamics, Layer 2 captures high-level trends
4. ~592K parameters: 198K (Layer 1) + 394K (Layer 2)
5. Dropout 0.3: Between layers for regularization

Looking Ahead: PyTorch's built-in LSTM uses batch normalization implicitly through careful initialization. However, adding explicit layer normalization can further stabilize training, especially for deep networks. The next section covers layer normalization integration.

With the two-layer design established, we now examine layer normalization for enhanced training stability.