Regularization: Ridge and Lasso

Learning Objectives

By the end of this section, you will be able to:

Where You'll Apply This: Preventing overfitting in any regression model, automatic feature selection, building interpretable models, weight decay in deep learning, and handling multicollinearity in high-dimensional data.

The Big Picture

Ordinary Least Squares (OLS) finds the coefficients that minimize the sum of squared errors. But what if this solution is too good—fitting the training data perfectly but failing to generalize? This is the problem of overfitting, and regularization is one of the most elegant solutions ever devised.

Historical Context

The Overfitting Problem

Consider what happens when we have many features relative to our sample size. OLS can find coefficients that fit the training data perfectly but fail dramatically on new data. The coefficients become unstable—small changes in the data cause wild swings in estimates.

Ridge Regression (L2 Regularization)

Ridge regression adds a penalty proportional to the sum of squared coefficients. This is equivalent to constraining the coefficients to lie within an L2 ball centered at the origin.

Mathematical Formulation

Let's break down each component:

Closed-Form Solution

One of Ridge regression's great advantages is that it has a closed-form solution, requiring only matrix inversion:

The key difference is the addition of $\lambda \mathbf{I}$ to the matrix$\mathbf{X}^T\mathbf{X}$. This has profound consequences:

• Invertibility: Even if $\mathbf{X}^T\mathbf{X}$ is singular (happens when $p > n$), adding $\lambda \mathbf{I}$makes it invertible.
• Numerical Stability: Small eigenvalues of $\mathbf{X}^T\mathbf{X}$become $\lambda_i + \lambda$, preventing division by near-zero values.
• Shrinkage: Coefficients are systematically shrunk toward zero, with the amount of shrinkage controlled by $\lambda$.

Shrinkage and Eigenvalues

To understand Ridge more deeply, express the solution using the singular value decompositionof $\mathbf{X}$. If $\mathbf{X} = \mathbf{U}\mathbf{D}\mathbf{V}^T$, then the Ridge coefficients are:

The term $\frac{d_j}{d_j^2 + \lambda}$ is the shrinkage factor. When $d_j$ is small (corresponding to directions of low variance in $\mathbf{X}$), the shrinkage is strong. When $d_j$ is large, shrinkage is mild.

Lasso Regression (L1 Regularization)

Lasso (Least Absolute Shrinkage and Selection Operator) uses the sum of absolute values of coefficients as the penalty. This seemingly small change has dramatic consequences.

Mathematical Formulation

Unlike Ridge, Lasso does not have a closed-form solution because the L1 norm is not differentiable at zero. We must use iterative algorithms like coordinate descent.

Why Lasso Produces Sparsity

The magic of Lasso lies in the soft thresholding operation. For a single variable, the Lasso solution has the form:

This formula reveals everything:

• If $|\hat{\beta}^{\text{OLS}}| < \lambda$, the coefficient is setexactly to zero. The feature is eliminated from the model.
• If $|\hat{\beta}^{\text{OLS}}| > \lambda$, the coefficient is shrunk by exactly $\lambda$ toward zero.
• The sign of the coefficient is preserved (positive stays positive, negative stays negative).

Geometric Interpretation

The difference between Ridge and Lasso becomes crystal clear when we view them geometrically. Both problems can be written as constrained optimization:

The regularized solution is found where the elliptical contours of the RSS(centered at the OLS solution) first touch the constraint region. For the smooth circle of Ridge, this almost always happens on the curved boundary—giving a non-sparse solution. For the diamond of Lasso, the contours often hit a corner, where one or more coordinates are exactly zero.

Interactive: Ridge vs Lasso Geometry

Explore how the different constraint shapes lead to different solutions. Adjust the OLS solution and constraint size to see when Lasso produces zeros at the corners.

Bias-Variance Tradeoff

Regularization is fundamentally about the bias-variance tradeoff. By introducing bias (shrinking coefficients away from their OLS values), we can achieve a larger reduction in variance, leading to lower overall prediction error.

As regularization strength $\lambda$ increases:

• Bias increases: We're forcing coefficients away from their "true" OLS values.
• Variance decreases: The estimates become less sensitive to the particular training sample.
• MSE has a U-shape: There's an optimal $\lambda$ where the total error is minimized.

Interactive: Bias-Variance Demo

Watch how regularization trades bias for variance. The thin lines show individual fits from different training samples (variance), while the thick lines show the average (bias is the deviation from the true line).

Regularization Paths

A regularization path shows how coefficients change as $\lambda$varies from 0 (OLS) to infinity (all zeros). This visualization is invaluable for understanding which features are most important and how they relate to each other.

Interactive: Regularization Path Explorer

Explore how Ridge and Lasso shrink coefficients differently. Notice how Lasso sets coefficients to zero while Ridge only approaches zero.

Selecting the Regularization Parameter

The regularization parameter $\lambda$ is a hyperparameterthat controls the bias-variance tradeoff. Choosing it well is crucial for good performance.

1. Cross-Validation: The gold standard. Split data into k folds, fit models for each $\lambda$ value on k-1 folds, evaluate on the held-out fold. Choose $\lambda$ with lowest average validation error.
2. 1-Standard-Error Rule: Instead of the minimum-error $\lambda$, choose the largest $\lambda$ within 1 SE of the minimum. This gives a simpler, more interpretable model with similar performance.
3. Information Criteria: AIC, BIC, and related criteria balance fit against complexity without requiring held-out data. Useful for very large datasets where CV is expensive.

Interactive: Cross-Validation for Lambda Selection

Explore how cross-validation finds the optimal regularization strength. Notice the U-shaped validation curve and how the 1-SE rule selects a more regularized model.

Elastic Net: Best of Both Worlds

What if you want the sparsity of Lasso but also need to handle groups of correlated features?Elastic Net combines both L1 and L2 penalties:

Deep Learning Connection

Regularization in linear models directly translates to deep learning. Understanding these connections illuminates why neural network training uses similar techniques.

# L2 regularization in PyTorch
optimizer = torch.optim.Adam(
    model.parameters(),
    lr=0.001,
    weight_decay=0.01  # This is lambda!
)

# Equivalent to adding lambda * sum(w^2) to the loss

Python Implementation

Here are complete implementations of Ridge and Lasso regression from scratch. Ridge uses the closed-form solution while Lasso requires coordinate descent.

When alpha=0, this reduces to OLS: (X'X)^(-1)X'y

S(rho, alpha) = sign(rho) * max(|rho| - alpha, 0)

1import numpy as np
2
3# Ridge Regression (L2 Regularization)
4class RidgeRegression:
5
6    def __init__(self, alpha=1.0):
7        self.alpha = alpha  # Regularization strength
8        self.coef_ = None
9        self.intercept_ = None
10
11    def fit(self, X, y):
12        # Add intercept column
13        n, p = X.shape
14        X_aug = np.column_stack([np.ones(n), X])
15
16        # Ridge closed-form: (X'X + alpha*I)^(-1) X'y
17        I = np.eye(p + 1)
18        I[0, 0] = 0  # Don't regularize intercept
19        XtX = X_aug.T @ X_aug
20        Xty = X_aug.T @ y
21        theta = np.linalg.solve(XtX + self.alpha * I, Xty)
22
23        # Extract intercept and coefficients
24        self.intercept_ = theta[0]
25        self.coef_ = theta[1:]
26        return self
27
28    def predict(self, X):
29        return self.intercept_ + X @ self.coef_
30
31
32# Lasso Regression (L1 Regularization)
33class LassoRegression:
34
35    def __init__(self, alpha=1.0, max_iter=1000, tol=1e-4):
36        self.alpha = alpha
37        self.max_iter = max_iter
38        self.tol = tol
39        self.coef_ = None
40        self.intercept_ = None
41
42    def fit(self, X, y):
43        n, p = X.shape
44
45        # Center the data (standard for Lasso)
46        X_mean = X.mean(axis=0)
47        y_mean = y.mean()
48        X_centered = X - X_mean
49        y_centered = y - y_mean
50
51        # Initialize coefficients
52        beta = np.zeros(p)
53
54        # Coordinate descent
55        for iteration in range(self.max_iter):
56            beta_old = beta.copy()
57
58            for j in range(p):
59                # Compute partial residual
60                residual = y_centered - X_centered @ beta
61                residual += X_centered[:, j] * beta[j]
62
63                # Compute rho_j
64                rho_j = X_centered[:, j] @ residual
65
66                # Soft thresholding
67                z_j = (X_centered[:, j] ** 2).sum()
68                if rho_j < -self.alpha * n:
69                    beta[j] = (rho_j + self.alpha * n) / z_j
70                elif rho_j > self.alpha * n:
71                    beta[j] = (rho_j - self.alpha * n) / z_j
72                else:
73                    beta[j] = 0  # Sparsity!
74
75            # Check convergence
76            if np.max(np.abs(beta - beta_old)) < self.tol:
77                break
78
79        self.coef_ = beta
80        self.intercept_ = y_mean - X_mean @ beta
81        return self
82
83    def predict(self, X):
84        return self.intercept_ + X @ self.coef_

Knowledge Check

Test your understanding of regularization with these questions:

Summary