Residual Connections

Learning Objectives

By the end of this section, you will:

1. Understand residual connections and their role in deep networks
2. Explain how residuals improve gradient flow
3. Combine layer normalization with residual connections
4. Assemble the complete attention block with all components

Why This Matters: Residual connections are essential for training deep networks. By providing a direct path for gradients, they prevent vanishing gradients and allow the network to learn incremental refinements rather than complete transformations. Every Transformer uses this pattern.

Residual Connection Concept

A residual connection adds the layer's input to its output, allowing the layer to learn a "residual" or difference.

Standard vs Residual

1Standard layer:
2  y = F(x)               ← Learn full transformation
3
4Residual layer:
5  y = x + F(x)           ← Learn delta (residual)
6      ↑   ↑
7      │   └── Learned transformation
8      └────── Skip connection (identity)

Why "Residual"?

The layer learns $F(x) = y - x$, the difference (residual) between desired output and input. This is often easier than learning the full transformation.

Gradient Flow Benefits

Residual connections create a "gradient highway" that prevents vanishing gradients.

Gradient Path Analysis

For $y = x + F(x)$, the gradient is:

The "1" term ensures gradients always flow, even if $\frac{\partial F}{\partial x} \approx 0$.

Deep Network Gradient

Through L residual layers:

This product contains many "1" terms, preventing the gradient from vanishing.

Comparison

Layer Norm + Residual

Residual connections are typically combined with layer normalization. Two common patterns exist.

Post-Norm (Original Transformer)

1x → Sublayer(x) → Add(x, ·) → LayerNorm → output
2
3y = LayerNorm(x + Sublayer(x))

Pre-Norm (Our Approach)

1x → LayerNorm → Sublayer → Add(x, ·) → output
2
3y = x + Sublayer(LayerNorm(x))

Comparison

Our choice: Post-Norm with careful initialization, as this is the standard in PyTorch's MultiheadAttention.

Complete Attention Block

Combining multi-head attention, residual connection, and layer normalization:

Block Structure

1Input: x (B, 30, 256)
2           │
3           ├─────────────────────────┐
4           ↓                         │
5  MultiHeadAttention(x, x, x)       │ (skip connection)
6           ↓                         │
7        Dropout                      │
8           ↓                         │
9          Add ←───────────────────────┘
10           ↓
11       LayerNorm
12           ↓
13      Output: (B, 30, 256)

Mathematical Expression

Where MHA is Multi-Head Attention with Q = K = V = x (self-attention).

Component Roles

Aggregation for Prediction

After the attention block, we have outputs for all 30 timesteps. For RUL prediction, we need a single vector. Common strategies:

• Mean pooling: $\bar{h} = \frac{1}{T}\sum_t h_t$
• Last timestep: $h_T$
• Learned pooling: Another attention over timesteps

Our model uses mean pooling over the attention output for simplicity and effectiveness.

Summary

In this section, we integrated residual connections with attention:

1. Residual connection: y = x + F(x), learns refinements
2. Gradient benefit: Direct path prevents vanishing gradients
3. Post-norm pattern: LayerNorm(x + Sublayer(x))
4. Complete block: MHA → Dropout → Add → LayerNorm

Looking Ahead: We have designed all components of the attention mechanism. The next section brings everything together with the complete PyTorch implementation, ready for integration with the CNN-BiLSTM backbone.

With all attention components designed, we now implement the complete attention module in PyTorch.