Dropout and Weight Decay

Introduction

In the previous section, we saw that overfitting happens when a model memorizes noise instead of learning patterns. Now we turn to the two most powerful weapons against overfitting: Dropout and Weight Decay. Together, these techniques are used in virtually every modern neural network, from small classifiers to GPT-4.

Each technique attacks overfitting from a different angle. Dropout is a stochastic method — it randomly disables parts of the network during training. Weight decay is a deterministic method — it continuously shrinks all weights toward zero. Despite their different mechanisms, both achieve the same goal: forcing the model to learn simpler, more generalizable representations.

Dropout: The Unreliable Team

Imagine you are training a team of 10 people for a critical presentation. Every morning, you randomly send 5 of them home — they cannot attend today's rehearsal. The remaining 5 must figure out how to cover all roles by themselves. Tomorrow, a different random half shows up. The day after, yet another combination.

At first, this seems like chaos. But after weeks of this, something remarkable happens: every team member becomes a generalist. Nobody can afford to hyper-specialize because they might not be there tomorrow. Everyone learns to handle the introduction, the data analysis, and the conclusions — just in case. When the full team finally reunites for the actual presentation, they are far more robust than a team where Alice always does slides and Bob always does analysis. If Alice gets sick on presentation day, the team can still function.

The Biological Parallel: Srivastava et al. (2014) drew an explicit analogy to sexual reproduction in biology. In asexual reproduction, the entire genome passes to offspring unchanged. In sexual reproduction, genes must work well with random combinations of genes from the other parent. This forces genes to be "good team players" — individually useful in many contexts, not dependent on specific gene partners. Dropout creates the same pressure on neurons.

How Dropout Works Mathematically

During training, each neuron's output is independently set to zero with probability $p$ (the drop rate). The surviving activations are then scaled up by $1/(1-p)$ to preserve the expected output magnitude. This is called inverted dropout.

Formally, given a layer's activation vector $\mathbf{h}$, the dropout operation is:

$\text{Training: } \quad \mathbf{m} \sim \text{Bernoulli}(1-p), \quad \tilde{\mathbf{h}} = \frac{\mathbf{m} \odot \mathbf{h}}{1-p}$

$\text{Inference: } \quad \tilde{\mathbf{h}} = \mathbf{h} \quad \text{(no modification)}$

The scaling by $1/(1-p)$ is the clever trick. Since each element of $\mathbf{m}$ is 1 with probability $(1-p)$ and 0 with probability $p$:

$\mathbb{E}[\tilde{h}_i] = \frac{(1-p) \cdot h_i}{1-p} = h_i$

The expected value of each output element is exactly the original activation — no scaling needed at test time. This property makes inverted dropout the standard implementation in PyTorch, TensorFlow, and every modern framework.

What Gets Dropped Where

Dropout Inside a Neural Network

The visualization below shows a small neural network with dropout applied. Toggle between training mode (random neurons are dropped) and inference mode (all neurons active). Adjust the dropout rate to see how aggressively the network is thinned:

Notice how at high dropout rates (0.7+), only a skeleton of the network remains. Each training step uses a different skeleton. A network with $n$ hidden units has $2^n$ possible sub-networks — dropout implicitly trains this exponential ensemble.

The Ensemble Interpretation

Why does randomly breaking a network make it better? The answer lies in ensemble learning. It is a well-known fact in machine learning that averaging the predictions of many different models produces better results than any single model. The problem is that training thousands of separate models is expensive.

Dropout gives us ensembles for free. Each dropout mask defines a different sub-network — a different "expert." With $n$ hidden units, there are $2^n$ possible masks, hence $2^n$ sub-networks sharing the same parameters. Each mini-batch trains a different sub-network. At test time, using the full network with scaled weights approximates the geometric mean of all these sub-networks' predictions — an ensemble of exponentially many models, trained with a single set of shared weights.

Gal & Ghahramani (2016, §3) made this ensemble view precise. Treat the network's weights as a random variable with a posterior $p(\theta \mid D)$ conditioned on the data $D$. Each dropout mask $m$ samples an approximate weight configuration $\theta_m = \theta \odot m$. Averaging predictions over many masks at test time \u2014 keeping dropout active during inference, repeating the forward pass, and averaging the outputs \u2014 is approximate Monte-Carlo Bayesian inference, $p(y \mid x, D) \approx \frac{1}{M} \sum_{i=1}^{M} f(x; \theta_{m_i})$. This is "MC Dropout", and it gives calibrated uncertainty estimates from any network you trained with dropout \u2014 useful for active learning, anomaly detection, medical diagnosis, and autonomous driving.

Implementing Dropout from Scratch

The implementation of dropout is surprisingly simple — just a binary mask and a scaling factor. The code below shows both the forward pass (masking and scaling) and the backward pass (same mask, same scaling). Click any line to see the exact values flowing through:

1import numpy as np
2
3def dropout_forward(x, drop_rate, training=True):
4    """Inverted dropout: scale during training, no-op at inference."""
5    if not training or drop_rate == 0.0:
6        return x, None
7
8    keep_prob = 1.0 - drop_rate
9
10    mask = np.random.binomial(1, keep_prob, size=x.shape)
11
12    out = (x * mask) / keep_prob
13
14    return out, mask
15
16def dropout_backward(grad_output, mask, drop_rate):
17    """Backprop through dropout: same mask, same scaling."""
18    keep_prob = 1.0 - drop_rate
19    return (grad_output * mask) / keep_prob
20
21# --- Example with concrete values ---
22np.random.seed(42)
23x = np.array([1.5, -0.8, 2.1, 0.3, -1.2, 0.7, 1.9, -0.5])
24
25print("Input x:", x)
26print("Sum(x):", x.sum().round(4))
27print()
28
29# Training: 50% dropout
30out_train, mask = dropout_forward(x, drop_rate=0.5, training=True)
31print("Mask:       ", mask)
32print("Train out:  ", out_train)
33print("Sum(out):   ", out_train.sum().round(4))
34print("E[sum(out)]:", x.sum().round(4), "(preserved!)")
35print()
36
37# Inference: no dropout
38out_test, _ = dropout_forward(x, drop_rate=0.5, training=False)
39print("Test out:   ", out_test)
40print("Sum(test):  ", out_test.sum().round(4))

The key insight: the scaling by $1/(1-p)$ during training means no modification is needed at test time. This is why it is called "inverted" dropout — the compensation happens during training rather than at inference, simplifying deployment.

Dropout in PyTorch with nn.Dropout

The NumPy implementation made the math explicit. In practice you reach for nn.Dropout, which handles the inverted scaling for you and switches itself off in eval mode.

1import torch
2import torch.nn as nn
3
4torch.manual_seed(0)
5
6# Define a tiny model with dropout between layers
7model = nn.Sequential(
8    nn.Linear(8, 16),
9    nn.ReLU(),
10    nn.Dropout(p=0.5),
11    nn.Linear(16, 4),
12)
13
14x = torch.ones(1, 8)  # fixed input
15
16# TRAINING MODE — dropout is active
17model.train()
18out_train_a = model(x)
19out_train_b = model(x)  # different! same input, different mask
20print("train pass 1:", out_train_a.detach().numpy().round(3))
21print("train pass 2:", out_train_b.detach().numpy().round(3))
22
23# EVALUATION MODE — dropout becomes the identity
24model.eval()
25out_eval_a = model(x)
26out_eval_b = model(x)
27print("eval  pass 1:", out_eval_a.detach().numpy().round(3))
28print("eval  pass 2:", out_eval_b.detach().numpy().round(3))
29
30# Verify expected magnitude: average of many train forwards ≈ eval output
31# (PyTorch's nn.Dropout uses inverted scaling, so this works automatically)
32n_runs = 1000
33model.train()
34train_avg = sum(model(x) for _ in range(n_runs)) / n_runs
35model.eval()
36eval_avg = model(x)
37print("avg train ≈ eval?", torch.allclose(train_avg, eval_avg, atol=0.1))

Weight Decay: The Tax on Complexity

Imagine every parameter in your model must pay a "tax" proportional to its magnitude. A weight of 5.0 pays more tax than a weight of 0.1. To justify its cost, a large weight must significantly reduce the loss — otherwise the tax pushes it back toward zero. Parameters that don't earn their keep get taxed into irrelevance.

Another way to think about it: every weight has an elastic band attached to the origin. The band pulls the weight toward zero with a force proportional to the weight's magnitude ($-\lambda w$). During training, the data gradient pulls the weight away from zero (to fit the data), while the elastic band pulls it back (to keep things simple). The equilibrium is a compromise between fitting and simplicity.

Occam's Razor in Code: Weight decay is the mathematical implementation of "prefer simpler explanations." A model with large weights is making bold, specific claims about the data. Weight decay penalizes boldness, preferring models that make gentle, conservative predictions — which tend to generalize better. Only parameters that genuinely reduce the loss survive the tax.

The Mathematics of Weight Decay

Weight decay adds a penalty term to the loss function that is proportional to the squared magnitude of all weights:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{data}} + \frac{\lambda}{2} \|\mathbf{w}\|_2^2 = \mathcal{L}_{\text{data}} + \frac{\lambda}{2} \sum_i w_i^2$

Taking the gradient with respect to $w$:

$\frac{\partial \mathcal{L}_{\text{total}}}{\partial w} = \frac{\partial \mathcal{L}_{\text{data}}}{\partial w} + \lambda w$

The SGD update becomes:

$w_{t+1} = w_t - \eta \left( \frac{\partial \mathcal{L}}{\partial w} + \lambda w_t \right) = (1 - \eta \lambda) w_t - \eta \frac{\partial \mathcal{L}}{\partial w}$

The factor $(1 - \eta \lambda)$ is the "decay" — each step, every weight shrinks by this fraction before the gradient update. With $\eta = 0.001$ and $\lambda = 0.01$, the multiplicative factor is $0.99999$, meaning each weight decays by 0.001% per step. Over thousands of steps, this gentle pull has a dramatic cumulative effect.

Bayesian Interpretation

L2 regularization has an elegant Bayesian interpretation: it is equivalent to placing a Gaussian prior on the weights centered at zero:

$p(\mathbf{w}) = \mathcal{N}(\mathbf{0}, \sigma^2 \mathbf{I}) \quad \Rightarrow \quad -\log p(\mathbf{w}) = \frac{1}{2\sigma^2} \|\mathbf{w}\|_2^2 + \text{const}$

Comparing with the L2 penalty $\frac{\lambda}{2} \|\mathbf{w}\|^2$, we get $\lambda = 1/\sigma^2$. A strong regularization ($\lambda$ large) corresponds to a tight prior ($\sigma$ small) — a strong belief that weights should be near zero. A weak regularization corresponds to a wide prior — willingness to let weights grow large if the data demands it.

The MAP (Maximum A Posteriori) estimate with this Gaussian prior is exactly the L2-regularized loss minimum. This is not coincidence — it is the same optimization problem viewed from two complementary perspectives: frequentist (penalty) and Bayesian (prior).

Geometric View: The Constraint Sphere

The L2 penalty $\|\mathbf{w}\|_2^2 \leq c$ constrains the weights to lie within a hypersphere centered at the origin. The regularization strength $\lambda$ controls the radius: larger $\lambda$ means a smaller sphere (tighter constraint on weight magnitude).

The visualization below shows this geometrically. The elliptical contours represent the data loss (where the model wants the weights to be), and the circle represents the L2 constraint (where regularization forces them). The optimum lies at the intersection — the best fit within the constraint:

The contour plot above shows the geometry of the constraint. The animation below shows the same effect in the time domain: a histogram of all model weights compressing toward zero as weight decay does its work. Drag $\lambda$ to feel how decay strength controls the speed of compression.

There is a deeper geometric insight from Goodfellow et al. (2016): weight decay rescales parameters along the eigenvectors of the Hessian. Parameters along directions with large eigenvalues (directions that strongly affect the loss) are barely shrunk. Parameters along directions with small eigenvalues (directions the loss is insensitive to) are aggressively shrunk toward zero. Weight decay selectively compresses the model's capacity in exactly the directions that don't matter for fitting the data.

Adam vs. AdamW: Why L2 \u2260 Weight Decay

For SGD, L2 regularization and weight decay are mathematically equivalent — they produce identical updates. But for adaptive optimizers like Adam, they are fundamentally different, and getting this wrong can significantly hurt training.

The problem: in Adam with L2 regularization, the regularization gradient $\lambda w$ gets fed into the adaptive moment estimates. This means it gets divided by $\sqrt{v_t}$ — the running average of squared gradients. Parameters with large gradient history (large $v_t$) receive weaker regularization, while parameters with small gradients receive disproportionately strong regularization. This is an unintended, parameter-dependent distortion.

AdamW (Loshchilov & Hutter, 2019) fixes this by applying weight decay directly to the weights, completely bypassing the adaptive scaling. The visualization below shows both optimizers converging on the same loss surface — notice how their trajectories and final positions differ:

What practitioners actually saw. Before AdamW, training transformers and ResNets with Adam plus an L2 penalty consistently underperformed SGD with momentum on image classification \u2014 faster initial convergence, worse final test error. Loshchilov & Hutter (2019, \u00a71) traced this gap to exactly the asymmetry derived above: Adam's per-parameter rescaling shrinks the L2 contribution most aggressively for parameters with the largest second-moment estimate, which is the opposite of what regularization should do. AdamW removed the asymmetry by decoupling the decay step, and the gap closed.

The code below implements both approaches so you can see exactly where the decoupling happens:

SGD + L2:  tensor([1.9480, -1.5085])
SGD + WD:  tensor([1.9480, -1.5085])
Equal?     True

1import torch
2import torch.nn as nn
3
4# --- SGD with L2 regularization (manual) ---
5def sgd_with_l2(params, grads, lr, lambda_):
6    """SGD + L2: gradient of penalty is added to loss gradient."""
7    updated = []
8    for w, g in zip(params, grads):
9        total_grad = g + lambda_ * w
10        w_new = w - lr * total_grad
11        updated.append(w_new)
12    return updated
13
14# --- SGD with weight decay (equivalent to L2 for SGD!) ---
15def sgd_with_weight_decay(params, grads, lr, lambda_):
16    """SGD + WD: decay is applied directly to weights."""
17    updated = []
18    for w, g in zip(params, grads):
19        w_new = (1 - lr * lambda_) * w - lr * g
20        updated.append(w_new)
21    return updated
22
23# --- AdamW (decoupled weight decay) ---
24def adamw_step(params, grads, m, v, t, lr, lambda_,
25               beta1=0.9, beta2=0.999, eps=1e-8):
26    """AdamW: weight decay is decoupled from adaptive gradient step."""
27    updated_params = []
28    updated_m = []
29    updated_v = []
30
31    for w, g, mi, vi in zip(params, grads, m, v):
32        mi_new = beta1 * mi + (1 - beta1) * g
33        vi_new = beta2 * vi + (1 - beta2) * g ** 2
34
35        m_hat = mi_new / (1 - beta1 ** t)
36        v_hat = vi_new / (1 - beta2 ** t)
37
38        adam_step = lr * m_hat / (torch.sqrt(v_hat) + eps)
39
40        w_new = (1 - lr * lambda_) * w - adam_step
41
42        updated_params.append(w_new)
43        updated_m.append(mi_new)
44        updated_v.append(vi_new)
45
46    return updated_params, updated_m, updated_v
47
48# --- Demonstrate the equivalence (SGD) and difference (Adam) ---
49torch.manual_seed(42)
50w = torch.tensor([2.0, -1.5])
51g = torch.tensor([0.5, 0.1])
52lr, lam = 0.1, 0.01
53
54# SGD + L2 vs SGD + WD: should be identical
55w_l2 = sgd_with_l2([w], [g], lr, lam)[0]
56w_wd = sgd_with_weight_decay([w], [g], lr, lam)[0]
57print(f"SGD + L2:  {w_l2}")
58print(f"SGD + WD:  {w_wd}")
59print(f"Equal?     {torch.allclose(w_l2, w_wd)}")

A practical consequence: with AdamW, the optimal weight decay is independent of the learning rate. You can tune them separately, which is a major simplification. With Adam+L2, changing the learning rate changes the effective regularization strength, making hyperparameter tuning a tangled mess.

The no_decay filter pattern

In every modern transformer training loop you will see a small helper that splits parameters into two AdamW groups: one decayed at the standard rate (typically $\lambda = 0.1$) and one with $\lambda = 0$. The no-decay group covers biases and LayerNorm parameters \u2014 decaying these one-dimensional scale/shift parameters tends to harm rather than help. The convention dates to GPT-3 (Brown et al., 2020, Appendix B) and is now the default in HuggingFace Transformers, fairseq, and reference implementations like nanoGPT.

1import torch
2import torch.nn as nn
3
4# What practitioners actually do: split params into decay vs no-decay groups.
5# This convention (excluding biases and LayerNorm scales from weight decay)
6# became standard via reference implementations like GPT-3 (Brown et al., 2020,
7# Appendix B) and is now the default in HuggingFace, fairseq, and nanoGPT.
8
9NO_DECAY = ("bias", "LayerNorm.weight", "ln_f.weight")
10
11def split_param_groups(model, weight_decay=0.1):
12    decay, no_decay = [], []
13    for name, param in model.named_parameters():
14        if not param.requires_grad:
15            continue
16        if any(nd in name for nd in NO_DECAY):
17            no_decay.append(param)
18        else:
19            decay.append(param)
20    return [
21        {"params": decay,    "weight_decay": weight_decay},
22        {"params": no_decay, "weight_decay": 0.0},
23    ]
24
25# Example with a tiny transformer-like block
26model = nn.Sequential(nn.Linear(64, 64), nn.LayerNorm(64), nn.Linear(64, 64))
27groups = split_param_groups(model, weight_decay=0.1)
28opt = torch.optim.AdamW(groups, lr=3e-4)
29
30print(f"decayed params:    {sum(p.numel() for p in groups[0]['params']):>5d}")
31print(f"non-decayed params: {sum(p.numel() for p in groups[1]['params']):>5d}")

Dropout and Weight Decay in Transformers

The original Transformer (Vaswani et al., 2017) uses dropout at three locations with rate $p = 0.1$:

1. After attention weights (post-softmax, pre-V multiply): prevents over-reliance on specific token-to-token relationships
2. After each sub-layer output (before residual addition): occasionally forces the model to rely purely on the skip connection
3. After embedding + positional encoding sum: regularizes the input representation

Plus label smoothing with $\varepsilon = 0.1$ as an output-level regularizer.

Why Modern LLMs Remove Dropout

A striking development: LLaMA, GPT-3, Mistral, PaLM, and most modern large language models use zero dropout. This seems counterintuitive — why remove a regularizer? The reasons are revealing:

Modern Transformer Training Recipe

Here is the configuration used by LLaMA (Touvron et al., 2023), representative of current best practices for large language models:

Notice the careful balance: no dropout, but strong weight decay (\u03bb=0.1). The implicit regularization from massive training data, large batch sizes, and weight decay replaces the explicit stochasticity of dropout. For smaller models or fine-tuning on limited data, dropout is still the go-to technique.

The implicit-regularization angle

At the scale of GPT-3 / LLaMA / Chinchilla there is also an implicit regularizer at work that has nothing to do with the techniques in this section \u2014 SGD's bias toward low-norm interpolators in the over-parameterized regime (Belkin et al., 2019; Nakkiran et al., 2020) and compute-optimal scaling (Hoffmann et al., 2022). See \u00a712.1 \u2014 Implicit Regularization at Scale for the full story.

Key Takeaways

1. Dropout randomly zeroes neuron outputs during training (rate $p$), scales survivors by $1/(1-p)$, and does nothing at inference. It implicitly trains an ensemble of $2^n$ sub-networks sharing the same parameters.
2. Weight decay shrinks all weights toward zero each step by a factor $(1 - \eta\lambda)$. It is equivalent to a Gaussian prior on the weights (Bayesian view) or constraining weights to a hypersphere (geometric view).
3. L2 \u2260 weight decay for Adam. In Adam+L2, the regularization gradient is distorted by adaptive scaling. AdamW decouples the decay, giving uniform regularization and independent hyperparameter tuning.
4. Not all parameters should be regularized. Weight decay is applied to weight matrices but NOT to biases, LayerNorm parameters, or embeddings.
5. Dropout prevents co-adaptation — neurons learn individually useful features rather than developing fragile co-dependencies.
6. Modern LLMs use weight decay without dropout. The combination of massive data, large batches, and AdamW (\u03bb=0.1) provides sufficient regularization. Dropout returns for fine-tuning on small datasets.
7. At scale, the implicit regularization from SGD's low-norm bias and Chinchilla-style data-rich training replaces explicit dropout (see \u00a712.1's Implicit Regularization at Scale).

Looking Ahead: In the next section, we will explore two more regularization techniques — Early Stopping and Data Augmentation — which control overfitting through training dynamics and data manipulation rather than model modification.

References

• Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. JMLR 15(56), 1929\u20131958.
• Gal, Y. & Ghahramani, Z. (2016). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. ICML 2016.
• Krogh, A. & Hertz, J. A. (1991). A Simple Weight Decay Can Improve Generalization. NIPS 1991 (NeurIPS 4).
• Loshchilov, I. & Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR 2019.
• Brown, T. B. et al. (2020). Language Models are Few-Shot Learners (GPT-3), Appendix B (parameter-group split convention). NeurIPS 2020.
• Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
• Goodfellow, I., Bengio, Y. & Courville, A. (2016). Deep Learning. MIT Press.
• Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.