The Learning Problem

Learning Objectives

By the end of this section, you will be able to:

1. Understand what "learning" means in the context of neural networks—it's optimization over a loss landscape
2. Describe the learning loop that drives all neural network training: forward pass → loss → backward pass → update
3. Visualize loss surfaces and understand why finding the minimum is challenging in high dimensions
4. Explain gradient descent as the fundamental algorithm for navigating loss landscapes
5. Connect loss functions to learning objectives and understand why different tasks need different loss functions
6. Recognize why backpropagation is essential—computing gradients efficiently is the key to training deep networks

Why This Matters: Before diving into the mechanics of backpropagation, you need to understand what we're trying to achieve. Learning is optimization. The loss function defines success. Gradients show us the way. This section builds the conceptual foundation for everything that follows.

What Does It Mean to Learn?

When we say a neural network "learns," we mean something very specific and mathematical. Unlike biological learning, which remains mysterious, machine learning has a precise definition:

Learning is the process of adjusting a model's parameters to minimize a measure of error on training data.

Let's unpack this definition. A neural network is a parameterized function $f(\mathbf{x}; \boldsymbol{\theta})$ that maps inputs $\mathbf{x}$ to outputs. The parameters $\boldsymbol{\theta}$ (weights and biases) determine the function's behavior. Learning means finding the parameter values that make the function produce the outputs we want.

The Key Insight

The breakthrough that enables machine learning is treating learning as an optimization problem:

Where:

• $\boldsymbol{\theta}^*$ is the optimal parameter vector we seek
• $\mathcal{L}(\boldsymbol{\theta})$ is the loss function—a scalar measure of how wrong our predictions are
• $\arg\min$ means "find the argument that minimizes"

This framing is powerful because optimization is a well-studied field. We can apply centuries of mathematical insights to make machines learn.

A Concrete Example

Consider training a network to predict house prices from features like square footage and location. Given training data $\{(\mathbf{x}^{(i)}, y^{(i)})\}_{i=1}^m$, we want parameters that make predictions $\hat{y}^{(i)} = f(\mathbf{x}^{(i)}; \boldsymbol{\theta})$ close to actual prices $y^{(i)}$.

Using mean squared error as our loss function:

Learning means finding $\boldsymbol{\theta}$ that makes this loss as small as possible.

The Learning Loop

All neural network training follows the same iterative pattern. Understanding this loop is crucial—it's the heartbeat of deep learning.

The Four Stages

1. Forward Pass

We feed input data through the network, layer by layer, computing predictions. Each layer applies its transformation:

The output of the final layer is our prediction $\hat{\mathbf{y}}$.

2. Loss Computation

We compare predictions to ground truth using a loss function. This produces a single scalar number that summarizes how wrong we are:

3. Backward Pass (Backpropagation)

Using the chain rule of calculus, we compute how each parameter contributed to the loss. We calculate gradients $\nabla_{\boldsymbol{\theta}} \mathcal{L}$—the direction of steepest increase in loss.

4. Parameter Update

We adjust parameters in the opposite direction of the gradient, reducing the loss:

Where $\eta$ is the learning rate—a crucial hyperparameter controlling step size.

The Optimization Perspective

To truly understand learning, we need to think about it geometrically. The loss function $\mathcal{L}(\boldsymbol{\theta})$ defines a loss surface over parameter space.

The Loss Surface

Imagine the parameters as coordinates in a high-dimensional space. For a network with 1 million parameters, this is a 1,000,000-dimensional space! At each point (parameter configuration), the loss function assigns a height (the loss value).

For visualization, consider just two parameters: the loss surface is then a 3D landscape. Hills represent high loss (bad predictions). Valleys represent low loss (good predictions). Training is like rolling a ball downhill to find the lowest point.

What Makes Optimization Hard?

In two dimensions, finding the minimum seems easy—just roll downhill! But real loss surfaces are much more complex:

Visualizing the Loss Landscape

The interactive visualization below shows a 3D loss surface. Watch as gradient descent navigates this landscape, following the negative gradient at each step. Try different learning rates to see their effects.

What You're Seeing

• The surface: A loss function with two parameters, showing a global minimum and some local structure
• The red ball: Current parameter position during optimization
• The yellow path: Trajectory taken by gradient descent
• The green marker: Location of the global minimum

Gradient Descent: Following the Slope

Gradient descent is the algorithm that enables neural network training. The idea is beautifully simple: the gradient tells us the direction of steepest ascent, so we walk in the opposite direction.

The Gradient

The gradient $\nabla_{\boldsymbol{\theta}} \mathcal{L}$ is a vector of partial derivatives:

Each component tells us how much the loss changes when we slightly perturb that parameter. The gradient points in the direction where loss increases fastest.

The Update Rule

Gradient descent iteratively updates parameters:

Where:

• $\boldsymbol{\theta}_t$ = parameters at iteration $t$
• $\eta$ = learning rate (step size)
• $\nabla_{\boldsymbol{\theta}} \mathcal{L}$ = gradient at current position

The Learning Rate Dilemma

The learning rate $\eta$ is critical:

Loss Functions: Measuring Error

The loss function is the objective we're optimizing. It quantifies how wrong our predictions are. Different tasks require different loss functions.

Mean Squared Error (MSE)

For regression (predicting continuous values):

Intuition: Penalizes predictions proportionally to the square of their error. Large errors are penalized much more than small ones.

Cross-Entropy Loss

For classification (predicting categories):

Intuition: Measures the difference between predicted probability distribution and true distribution. Heavily penalizes confident wrong predictions.

Binary Cross-Entropy

For binary classification:

Why Different Loss Functions?

The Statistical Learning Perspective

So far we've discussed fitting training data. But the real goal is generalization—performing well on new, unseen data.

The True Objective

We actually want to minimize the expected risk over the data distribution:

But we don't know the true data distribution $P_{\text{data}}$! We only have samples.

Empirical Risk Minimization

Instead, we minimize the empirical risk—the average loss over our training set:

This works when training data is representative of the true distribution. The gap between training loss and test loss tells us how well we've generalized.

The Bias-Variance Tradeoff

Two sources of error in learning:

• Bias: Error from the model being too simple (underfitting). A linear model can't learn XOR, no matter how much data you have.
• Variance: Error from the model being too sensitive to training data (overfitting). A very complex model might memorize noise.

Deep learning mitigates this through:

1. High capacity models (low bias)
2. Regularization techniques (controlled variance)
3. Massive datasets (reduces overfitting)
4. Careful architecture design (inductive biases)

The Challenge: Computing Gradients

We've established that gradient descent requires gradients. But how do we compute $\nabla_{\boldsymbol{\theta}} \mathcal{L}$ for a neural network with millions of parameters?

Three Approaches

1. Numerical Differentiation

Approximate derivatives using finite differences:

Problem: Requires 2 forward passes per parameter. For 1 million parameters, that's 2 million forward passes per gradient! Completely impractical.

2. Symbolic Differentiation

Apply calculus rules to derive gradient expressions analytically.

Problem: Expression size grows exponentially with network depth. Memory-intensive and computationally explosive.

3. Automatic Differentiation (Backpropagation)

Compute gradients through reverse-mode automatic differentiation—the famous backpropagation algorithm.

Advantage: Computes all gradients in essentially the same time as one forward pass. This is the breakthrough that makes deep learning possible!

Implementation Preview

Here's a preview of how these concepts translate to PyTorch code. We'll build this understanding fully in subsequent sections.

This network: 10 inputs → 64 → 32 → 1 output

lr=0.01 means steps of size 0.01 × gradient

1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5# 1. Define a simple model (parameterized function)
6model = nn.Sequential(
7    nn.Linear(10, 64),
8    nn.ReLU(),
9    nn.Linear(64, 32),
10    nn.ReLU(),
11    nn.Linear(32, 1)
12)
13
14# 2. Define the loss function (what we're optimizing)
15loss_fn = nn.MSELoss()
16
17# 3. Define the optimizer (how we update parameters)
18optimizer = optim.SGD(model.parameters(), lr=0.01)
19
20# The Learning Loop
21for epoch in range(num_epochs):
22    for batch_x, batch_y in dataloader:
23        # Forward pass: compute predictions
24        predictions = model(batch_x)
25
26        # Compute loss
27        loss = loss_fn(predictions, batch_y)
28
29        # Backward pass: compute gradients
30        optimizer.zero_grad()  # Clear previous gradients
31        loss.backward()        # Backpropagation!
32
33        # Update parameters
34        optimizer.step()       # θ ← θ - η∇L
35
36    print(f"Epoch {epoch}: Loss = {loss.item():.4f}")

Summary

This section established the conceptual foundation for understanding backpropagation and neural network training.

Key Concepts

The Big Picture

Learning is optimization. We have a parameterized model, a loss function that measures error, and an update rule (gradient descent) that adjusts parameters to reduce error. The key challenge is computing gradients efficiently—this is what backpropagation solves.

Coming Up: In the next sections, we'll dive deep into gradient descent variants, computational graphs, and the backpropagation algorithm itself. You'll implement backprop from scratch and understand exactly how modern frameworks like PyTorch compute gradients automatically.

Exercises

Conceptual Questions

1. Why is the loss function always a scalar? What would happen if it were a vector?
2. Explain why gradient descent moves in the negative gradient direction. What would happen if we moved in the positive gradient direction?
3. In the loss surface visualization, the path taken by gradient descent isn't a straight line to the minimum. Why does it curve?
4. What's the difference between batch gradient descent (using all training data), stochastic gradient descent (using one example), and mini-batch gradient descent (using a subset)?

Mathematical Exercises

1. For the loss function $\mathcal{L}(w, b) = (wx + b - y)^2$ with a single data point $(x, y) = (2, 5)$, compute the gradient $[\partial \mathcal{L}/\partial w, \partial \mathcal{L}/\partial b]$ at $(w, b) = (1, 1)$.
2. Show that for MSE loss, the gradient with respect to a prediction $\hat{y}$ is $\frac{2}{m}(\hat{y} - y)$.
3. Prove that gradient descent converges to the minimum for a convex quadratic function $f(x) = ax^2$ with $a > 0$, given appropriate learning rate.

Coding Exercises

1# Exercise: Implement gradient descent for linear regression from scratch
2# Given: Data points, initial weights, learning rate, number of iterations
3# Goal: Find optimal weights that minimize MSE
4
5import numpy as np
6
7# Generate data
8np.random.seed(42)
9X = np.random.randn(100, 1)
10y = 2 * X + 0.5 + 0.1 * np.random.randn(100, 1)
11
12# TODO: Implement gradient descent
13def gradient_descent(X, y, lr=0.1, n_iters=100):
14    w = np.zeros((1, 1))
15    b = 0.0
16    m = len(X)
17    losses = []
18
19    for i in range(n_iters):
20        # Forward pass: compute predictions
21        y_pred = # TODO
22
23        # Compute loss
24        loss = # TODO
25        losses.append(loss)
26
27        # Compute gradients
28        dw = # TODO
29        db = # TODO
30
31        # Update parameters
32        w = # TODO
33        b = # TODO
34
35    return w, b, losses
36
37# Test your implementation
38w_final, b_final, losses = gradient_descent(X, y)
39print(f"Learned: w = {w_final[0,0]:.3f}, b = {b_final:.3f}")
40print(f"True:    w = 2.0, b = 0.5")

You now understand the fundamental learning problem that backpropagation solves. In the next section, we'll explore gradient descent variants—the practical algorithms that make training neural networks efficient.