Overfitting and Regularization

Learning Objectives

By the end of this section, you will be able to:

1. Detect overfitting: Use training and validation curves to diagnose overfitting and underfitting in your models
2. Tune regularization hyperparameters: Find optimal values for weight decay, dropout rate, and early stopping patience
3. Combine techniques effectively: Know which regularization methods work well together and which conflict
4. Debug regularization problems: Diagnose under-regularization, over-regularization, and technique-specific issues

The Big Picture

You've learned what regularization techniques do (Chapter 5). Now the question is: how do you actually use them in practice?

Regularization is not just a matter of adding dropout or weight decay. It's about:

1. Detection: Recognizing when your model is overfitting (or underfitting)
2. Selection: Choosing which techniques to apply for your architecture
3. Tuning: Finding the right hyperparameter values
4. Combination: Making multiple techniques work together
5. Debugging: Fixing problems when regularization isn't working

Detecting Overfitting

Explore how model complexity affects fitting. Increase the polynomial degree to see how the model can fit training points perfectly while losing the ability to generalize:

Recognizing the Patterns

The primary tool for detecting overfitting is comparing training and validation loss curves over time:

Healthy Training

• Both losses decrease together
• The gap between them remains small and stable
• Both eventually plateau at similar values

Overfitting

• Training loss continues to decrease
• Validation loss decreases initially, then starts increasing
• The gap between them grows larger over time

Underfitting

• Both losses plateau at high values
• Neither loss shows significant improvement
• The model lacks capacity to learn the pattern

The Bias-Variance Tradeoff

The bias-variance tradeoff is fundamental to understanding why regularization works. Explore it interactively:

The key insight: optimal generalization occurs at a model complexity where the sum of bias and variance is minimized—not where either one is minimized individually.

Regularization Strategy

You know what regularization techniques do from Chapter 5. Now let's focus on the practical question: how do you tune them?

Tuning Weight Decay

Weight decay (L2 regularization) is almost always beneficial. The question is finding the right strength:

Tuning Dropout

Dropout rate depends on layer type and model capacity:

Interactive: Dropout Visualization

See how dropout randomly disables neurons during training. Toggle between training and inference modes to see the difference:

Early Stopping

Early stopping is a simple but effective regularization strategy: stop training before the model overfits.

The Algorithm

1. Monitor validation loss after each epoch
2. Track the best validation loss seen so far and save that model checkpoint
3. If validation loss doesn't improve for patience consecutive epochs, stop training
4. Restore the model weights from the best checkpoint

Why It Works

Early stopping is a form of implicit regularization. As training progresses:

1. Early epochs: Model learns general patterns (bias decreases, variance stays low)
2. Middle epochs: Model reaches optimal generalization point
3. Late epochs: Model starts memorizing training data (bias stays low, variance increases)

By stopping at the optimal point, we prevent the variance from increasing while maintaining the learned general patterns.

Interactive: Early Stopping

Watch a training run and see how early stopping detects the optimal point to stop training:

Early Stopping in Practice

Early stopping prevents overfitting by stopping training when validation loss stops improving. The key decisions are:

Combining Regularization Techniques

Regularization techniques can be combined, but some combinations work better than others:

Complementary Combinations

Conflicting Combinations

Recommended Baseline Setup

1# Solid baseline for most tasks
2optimizer = optim.AdamW(
3    model.parameters(),
4    lr=1e-3,
5    weight_decay=1e-2  # Start here, tune if needed
6)
7
8# Architecture includes appropriate normalization
9# (BatchNorm for CNNs, LayerNorm for Transformers)
10
11# Moderate dropout in FC layers
12nn.Dropout(0.3)  # Not too aggressive
13
14# Early stopping as safety net
15early_stopping = EarlyStopping(patience=10)
16
17# Data augmentation for image tasks
18transforms.Compose([
19    transforms.RandomHorizontalFlip(),
20    transforms.RandomCrop(32, padding=4),
21    transforms.ToTensor(),
22])

Debugging Regularization Issues

When regularization isn't working as expected, use this diagnostic guide:

Problem: Still Overfitting

Symptoms: Large train/val gap, val loss increasing while train loss decreases.

1. Increase weight decay (try 10x higher)
2. Add or increase dropout rate
3. Add more data augmentation
4. Reduce model capacity (fewer layers/neurons)
5. Lower early stopping patience

Problem: Underfitting

Symptoms: Both train and val loss high, neither improving.

1. Reduce weight decay (try 10x lower)
2. Reduce or remove dropout
3. Increase model capacity
4. Train longer (increase patience)
5. Check if learning rate is too low

Problem: Training Unstable

Symptoms: Loss spikes, NaN values, erratic behavior.

1. Check BatchNorm: are you calling model.train()/model.eval()?
2. Reduce learning rate
3. Check for gradient explosion (add gradient clipping)
4. Ensure batch size is large enough for BatchNorm (at least 16-32)

PyTorch Implementation

L2 Regularization (Weight Decay)

1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5# Method 1: Using weight_decay in optimizer
6model = nn.Sequential(
7    nn.Linear(784, 256),
8    nn.ReLU(),
9    nn.Linear(256, 10)
10)
11
12# Add L2 regularization via weight_decay
13optimizer = optim.SGD(
14    model.parameters(),
15    lr=0.01,
16    weight_decay=1e-4  # L2 regularization strength
17)
18
19# Method 2: AdamW for proper weight decay with Adam
20optimizer = optim.AdamW(
21    model.parameters(),
22    lr=0.001,
23    weight_decay=0.01  # Decoupled weight decay
24)
25
26# Method 3: Manual L2 regularization
27def train_with_manual_l2(model, data_loader, loss_fn, optimizer, l2_lambda):
28    model.train()
29    for inputs, targets in data_loader:
30        optimizer.zero_grad()
31        outputs = model(inputs)
32
33        # Compute data loss
34        data_loss = loss_fn(outputs, targets)
35
36        # Compute L2 penalty manually
37        l2_penalty = sum(p.pow(2).sum() for p in model.parameters())
38
39        # Total loss
40        total_loss = data_loss + l2_lambda * l2_penalty
41
42        total_loss.backward()
43        optimizer.step()

Dropout

1import torch.nn as nn
2
3class MLPWithDropout(nn.Module):
4    def __init__(self, input_size, hidden_size, num_classes, dropout_rate=0.5):
5        super().__init__()
6        self.model = nn.Sequential(
7            nn.Linear(input_size, hidden_size),
8            nn.ReLU(),
9            nn.Dropout(dropout_rate),  # Dropout after activation
10
11            nn.Linear(hidden_size, hidden_size),
12            nn.ReLU(),
13            nn.Dropout(dropout_rate),
14
15            nn.Linear(hidden_size, num_classes)  # No dropout before output
16        )
17
18    def forward(self, x):
19        return self.model(x)
20
21# Training: dropout is active
22model = MLPWithDropout(784, 256, 10, dropout_rate=0.5)
23model.train()  # Enables dropout
24output = model(input_tensor)  # Some neurons are dropped
25
26# Inference: dropout is disabled
27model.eval()  # Disables dropout
28output = model(input_tensor)  # All neurons active
29
30# For CNNs: use Dropout2d (drops entire feature maps)
31class CNNWithDropout(nn.Module):
32    def __init__(self):
33        super().__init__()
34        self.features = nn.Sequential(
35            nn.Conv2d(3, 64, 3, padding=1),
36            nn.ReLU(),
37            nn.Dropout2d(0.2),  # Spatial dropout for conv layers
38            nn.MaxPool2d(2),
39
40            nn.Conv2d(64, 128, 3, padding=1),
41            nn.ReLU(),
42            nn.Dropout2d(0.2),
43            nn.MaxPool2d(2),
44        )

Early Stopping

1class EarlyStopping:
2    """Early stopping to prevent overfitting."""
3
4    def __init__(self, patience=10, min_delta=0.0, restore_best=True):
5        self.patience = patience
6        self.min_delta = min_delta
7        self.restore_best = restore_best
8        self.best_loss = float('inf')
9        self.best_weights = None
10        self.epochs_without_improvement = 0
11
12    def __call__(self, val_loss, model):
13        """Returns True if training should stop."""
14        if val_loss < self.best_loss - self.min_delta:
15            # Improvement found
16            self.best_loss = val_loss
17            self.best_weights = model.state_dict().copy()
18            self.epochs_without_improvement = 0
19            return False
20        else:
21            # No improvement
22            self.epochs_without_improvement += 1
23            if self.epochs_without_improvement >= self.patience:
24                if self.restore_best and self.best_weights is not None:
25                    model.load_state_dict(self.best_weights)
26                return True  # Stop training
27            return False
28
29# Usage in training loop
30early_stopping = EarlyStopping(patience=10)
31
32for epoch in range(max_epochs):
33    train_loss = train_one_epoch(model, train_loader, optimizer)
34    val_loss = validate(model, val_loader)
35
36    print(f"Epoch {epoch}: train_loss={train_loss:.4f}, val_loss={val_loss:.4f}")
37
38    if early_stopping(val_loss, model):
39        print(f"Early stopping triggered at epoch {epoch}")
40        break
41
42# Model now has best weights restored

Summary

This section covered practical regularization strategies for training neural networks. Key takeaways:

Quiz

Test your understanding of overfitting and regularization:

Exercises

Diagnostic Scenarios

1. Scenario: Your model has 99% training accuracy but only 60% validation accuracy. The validation loss started increasing after epoch 5. What's happening, and what steps would you take (in order of priority)?
2. Scenario: Both your training and validation loss are stuck at high values (neither improving after 50 epochs). What might be wrong, and how would you diagnose it?
3. Scenario: You added dropout(0.5) to all layers, and now both train and validation accuracy dropped significantly. What went wrong?
4. Scenario: Your model works perfectly in training but gives random predictions during inference. What's the most likely cause?

Hands-On Experiments

1. Weight decay sweep: Train a CNN on CIFAR-10 with weight_decay values (0, 1e-5, 1e-4, 1e-3, 1e-2). Plot train/val curves for each. Identify which value gives the best validation accuracy.
2. Regularization combination: Compare three setups: (a) weight decay only, (b) dropout only, (c) both. Measure final validation accuracy and the train/val gap for each.
3. Early stopping patience: Train with patience values (3, 5, 10, 20, 50). Record at which epoch training stopped and the final validation loss. Find the sweet spot.
4. train() vs eval() experiment: Train a model with BatchNorm. Run inference both with and without calling model.eval(). Compare the predictions and accuracy.

In the next section, we'll explore hyperparameter tuning—systematic approaches to finding optimal values for learning rate, regularization strength, and other hyperparameters.