Building a CNN from Scratch

From Pixels to Predictions

In Chapter 13, we mastered the individual pieces: convolution operations, kernel mechanics, multi-channel filtering, and the output size formula. Now it's time to assemble these pieces into a complete, working CNN that can look at a handwritten digit and tell you what number it is.

We will build a CNN that classifies MNIST digits — 28×28 grayscale images of handwritten numbers 0 through 9. By the end of this section, you will have a network that achieves 99% accuracy with just two convolutional layers and about 207,000 parameters.

The CNN Recipe: A CNN is not just convolutions. It is a carefully designed pipeline: convolution (detect patterns) → activation (add non-linearity) → pooling (compress spatial info) → repeat → flatten (reshape for classification) → fully-connected layers (make the final decision). Each piece has a specific job.

But before we can build this pipeline, we need one crucial piece that Chapter 13 introduced but did not fully explore: pooling.

The Missing Piece: Pooling

After a convolutional layer detects features across the image, we face a problem: the feature maps are the same spatial size as the input (when using padding). A 28×28 image produces 28×28 feature maps. If we stack many conv layers, the computational cost grows rapidly. We need a way to shrink the spatial dimensions while keeping the important information.

That is exactly what pooling does. It slides a window across each feature map and summarizes each region with a single value. The two most common types are:

• Max Pooling: Keep the maximum value in each window. This preserves the strongest activation — if a vertical edge was detected somewhere in a 2×2 region, max pooling keeps that detection regardless of its exact position.
• Average Pooling: Compute the mean of all values in the window. This creates a smooth summary of activations, useful as a final aggregation layer.

Why Pooling Matters

Pooling provides three critical benefits:

1. Spatial reduction: A 2×2 pooling with stride 2 halves both height and width, reducing the number of values by 4×. This means the next conv layer has 4× fewer positions to process.
2. Translation invariance: If a feature shifts by 1 pixel in the input, max pooling often produces the same output. The network cares about what was detected, not exactly where.
3. Larger receptive field: After pooling, each position in the next layer's feature map “sees” a larger region of the original image. This enables higher layers to detect larger-scale patterns.

The Pooling Formula

The output size formula for pooling is the same as for convolution (no padding is typically used):

$O = \lfloor \frac{I - K}{S} \rfloor + 1$

For the standard 2×2 max pooling with stride 2: $O = \lfloor \frac{I - 2}{2} \rfloor + 1 = \frac{I}{2}$. So a 28×28 feature map becomes 14×14, and a 14×14 becomes 7×7.

Explore max pooling and average pooling interactively:

Max Pooling from Scratch

Let's implement max pooling ourselves to fully understand the mechanics. We will process a 4×4 feature map with a 2×2 pooling window and stride 2:

  col0  col1  col2  col3
    1     3     2     4
    5     6     1     2
    7     2     3     1
    4     8     5     6

  col0  col1  col2  col3
    1     3     2     4
    5     6     1     2
    7     2     3     1
    4     8     5     6

[[0.0, 0.0],
 [0.0, 0.0]]

[[1, 3],
 [5, 6]]

[[2, 4],
 [1, 2]]

[[7, 2],
 [4, 8]]

[[3, 1],
 [5, 6]]

[[6.0, 4.0],
 [8.0, 6.0]]

[[6.0, 4.0],
 [8.0, 6.0]]

[[6. 4.]
 [8. 6.]]

1import numpy as np
2
3# A 4x4 feature map (output from a conv layer)
4feature_map = np.array([
5    [1, 3, 2, 4],
6    [5, 6, 1, 2],
7    [7, 2, 3, 1],
8    [4, 8, 5, 6]
9])
10
11def max_pool_2d(x, pool_size=2, stride=2):
12    H, W = x.shape
13    H_out = (H - pool_size) // stride + 1
14    W_out = (W - pool_size) // stride + 1
15    output = np.zeros((H_out, W_out))
16
17    for i in range(H_out):
18        for j in range(W_out):
19            r = i * stride
20            c = j * stride
21            window = x[r:r+pool_size, c:c+pool_size]
22            output[i, j] = np.max(window)
23
24    return output
25
26result = max_pool_2d(feature_map)
27print(result)
28# [[6. 4.]
29#  [8. 6.]]

Max Pooling in PyTorch

PyTorch provides $\texttt{nn.MaxPool2d}$ which handles batches, multiple channels, and GPU acceleration:

Why ReLU, Not tanh or Sigmoid

Before we assemble the network we need to answer a practical question: which activation function goes between the convolution and the pooling? The 1998 LeNet paper used $\tanh$. Modern CNNs almost universally use $\mathrm{ReLU}(x) = \max(0, x)$. What changed?

The Vanishing-Gradient Problem of Smooth Squashers

Both $\tanh$ and $\sigma(x) = 1/(1+e^{-x})$ are S-shaped “squashers” that saturate at the extremes. When the pre-activation $x$ is large in magnitude, the derivative is almost zero:

In a 10-layer network using $\tanh$, gradients propagating backward through the chain rule multiply ten derivatives, each typically much less than 1. The gradient arriving at the earliest layer can be $10^{-10}$ of the loss gradient — effectively zero. The early filters never learn. This is the vanishing-gradient problem, documented by Glorot & Bengio (2010) and solved architecturally in several ways; ReLU was the simplest.

Nair & Hinton (2010) and Glorot, Bordes & Bengio (2011) showed that ReLU trains deeper networks faster. Its gradient is either exactly 0 (for $x < 0$) or exactly 1 (for $x > 0$). There is no gradual decay. When the signal is “on”, the gradient flows through untouched.

The dead-ReLU caveat. If a neuron's pre-activation stays negative for every input, the gradient is always zero and its weights never update — a “dead” neuron. Two common escape hatches: Leaky ReLU ($f(x) = \max(0.01x, x)$) and He initialisation (variance scaled by $2/n_{\text{in}}$) which tunes initial weights so most neurons start “alive”. We use plain ReLU here because MNIST is forgiving; He initialisation is a one-line fix for larger networks.

Our CNN Blueprint

Now we have all the building blocks: convolution (Chapter 13), activation functions (Chapter 5), pooling (above), and fully-connected layers (Chapter 10). Let's assemble them into a complete CNN for digit classification.

Click any layer in the diagram below to see exactly what it does and how many parameters it has:

Dimension Tracking: Every Tensor Shape

Understanding exactly how the tensor shape transforms at each layer is critical. Here is the complete flow for a single image:

Total: 206,922 parameters. Notice that the FC1 layer alone accounts for 97% of all parameters. This is a common pattern in CNNs — the convolutional layers are parameter-efficient (weight sharing across spatial positions), but the first fully-connected layer creates a large weight matrix.

The PyTorch Implementation

Let's build this CNN in PyTorch using $\texttt{nn.Module}$. The pattern is the same we learned in Chapter 10 for MLPs: define layers in \texttt{__init__}, wire them together in $\texttt{forward}$.

1import torch
2import torch.nn as nn
3
4class DigitCNN(nn.Module):
5    def __init__(self):
6        super().__init__()
7        # Conv Block 1: detect edges and simple textures
8        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, padding=1)
9        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
10
11        # Conv Block 2: combine edges into shapes
12        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1)
13        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
14
15        # Classification Head
16        self.fc1 = nn.Linear(32 * 7 * 7, 128)
17        self.dropout = nn.Dropout(0.25)
18        self.fc2 = nn.Linear(128, 10)
19
20    def forward(self, x):
21        # x: [batch, 1, 28, 28]
22        x = torch.relu(self.conv1(x))   # -> [batch, 16, 28, 28]
23        x = self.pool1(x)                # -> [batch, 16, 14, 14]
24        x = torch.relu(self.conv2(x))   # -> [batch, 32, 14, 14]
25        x = self.pool2(x)                # -> [batch, 32, 7, 7]
26        x = x.view(x.size(0), -1)       # -> [batch, 1568]
27        x = torch.relu(self.fc1(x))     # -> [batch, 128]
28        x = self.dropout(x)              # -> [batch, 128]
29        x = self.fc2(x)                  # -> [batch, 10]
30        return x
31
32model = DigitCNN()
33total = sum(p.numel() for p in model.parameters())
34print(f"Total parameters: {total:,}")
35# Total parameters: 206,922

Key Design Decisions

Training on MNIST

Now let's train our CNN on the MNIST dataset. The training loop follows the same pattern from Chapter 11: load data → forward pass → compute loss → backward pass → update weights. The only difference is that we're now feeding images through a CNN instead of flat vectors through an MLP.

1import torch.optim as optim
2from torchvision import datasets, transforms
3from torch.utils.data import DataLoader
4
5# Step 1: Prepare MNIST data
6transform = transforms.Compose([
7    transforms.ToTensor(),
8    transforms.Normalize((0.1307,), (0.3081,))
9])
10train_data = datasets.MNIST(
11    'data', train=True, download=True, transform=transform
12)
13test_data = datasets.MNIST(
14    'data', train=False, transform=transform
15)
16train_loader = DataLoader(train_data, batch_size=64, shuffle=True)
17test_loader = DataLoader(test_data, batch_size=1000)
18
19# Step 2: Set up model, loss, optimizer
20model = DigitCNN()
21criterion = nn.CrossEntropyLoss()
22optimizer = optim.Adam(model.parameters(), lr=0.001)
23
24# Step 3: Train for 3 epochs
25for epoch in range(3):
26    model.train()
27    running_loss, correct, total = 0.0, 0, 0
28
29    for images, labels in train_loader:
30        # Forward pass
31        outputs = model(images)
32        loss = criterion(outputs, labels)
33
34        # Backward pass + update
35        optimizer.zero_grad()
36        loss.backward()
37        optimizer.step()
38
39        # Track metrics
40        running_loss += loss.item()
41        _, predicted = outputs.max(1)
42        correct += predicted.eq(labels).sum().item()
43        total += labels.size(0)
44
45    # Evaluate on test set
46    model.eval()
47    test_correct, test_total = 0, 0
48    with torch.no_grad():
49        for images, labels in test_loader:
50            outputs = model(images)
51            _, predicted = outputs.max(1)
52            test_correct += predicted.eq(labels).sum().item()
53            test_total += labels.size(0)
54
55    train_acc = 100.0 * correct / total
56    test_acc = 100.0 * test_correct / test_total
57    avg_loss = running_loss / len(train_loader)
58    print(f"Epoch {epoch+1}: Loss={avg_loss:.4f}  "
59          f"Train={train_acc:.1f}%  Test={test_acc:.1f}%")
60
61# Epoch 1: Loss=0.1842  Train=94.3%  Test=98.2%
62# Epoch 2: Loss=0.0571  Train=98.2%  Test=98.8%
63# Epoch 3: Loss=0.0392  Train=98.7%  Test=99.0%

Understanding the Results

Several things stand out:

1. Rapid learning: After just 1 epoch (one pass through 60,000 images), the model already achieves 94% training accuracy. The convolutional structure makes learning dramatically easier than a fully-connected network.
2. Test > Train accuracy: Notice that test accuracy (98.2%) exceeds training accuracy (94.3%) in epoch 1. This is because dropout is active during training (randomly disabling 25% of neurons) but disabled during testing. The model is actually more capable than its training performance suggests.
3. 99% in 3 epochs: Our tiny 207K-parameter CNN reaches 99% test accuracy on MNIST. For context, a simple logistic regression achieves ~92% and a fully-connected MLP achieves ~97%. The convolutional structure provides a meaningful advantage on image data.

Why CNNs Beat MLPs on Images: An MLP treats each pixel independently — pixel (0,0) has no special relationship to pixel (0,1). A CNN exploits spatial locality (nearby pixels are related) and weight sharing (the same filter scans every position). This inductive bias matches the structure of images, so the network learns more from less data.

What the CNN Learned

The most fascinating part of training a CNN is examining what the filters actually learned. Remember: we never told the network about edge detection, Sobel filters, or any image processing concept. We only showed it digits and labels. Let's see what gradient descent discovered on its own.

  col0   col1   col2
-0.18   0.02   0.31
-0.27   0.05   0.34
-0.15   0.01   0.28

  col0    col1    col2
 0.22    0.25    0.19
 0.01   -0.03    0.02
-0.20   -0.28   -0.17

[[-1, 0, 1],
 [-2, 0, 2],
 [-1, 0, 1]]

[[-0.18, 0.02, 0.31],
 [-0.27, 0.05, 0.34],
 [-0.15, 0.01, 0.28]]

1# After training, extract what conv1 learned
2filters = model.conv1.weight.data   # shape: [16, 1, 3, 3]
3print(f"Layer 1: {filters.shape[0]} learned filters")
4
5# Filter 0: resembles a vertical edge detector
6print("\nFilter 0 (vertical edges):")
7print(filters[0, 0].numpy().round(2))
8# [[-0.18  0.02  0.31]
9#  [-0.27  0.05  0.34]
10#  [-0.15  0.01  0.28]]
11
12# Filter 5: resembles a horizontal edge detector
13print("\nFilter 5 (horizontal edges):")
14print(filters[5, 0].numpy().round(2))
15# [[ 0.22  0.25  0.19]
16#  [ 0.01 -0.03  0.02]
17#  [-0.20 -0.28 -0.17]]
18
19# Compare with the Sobel kernel from Chapter 13!
20print("\nSobel vertical edge kernel:")
21print("[[-1, 0, 1],")
22print(" [-2, 0, 2],")
23print(" [-1, 0, 1]]")
24
25# The CNN discovered edge detectors on its own!
26# No one told it about Sobel — it learned from data.

The Feature Hierarchy

The two convolutional layers form a feature hierarchy — each layer builds on the patterns detected by the previous one:

This hierarchy is why CNNs work so well. Layer 1 learns the same edge detectors that neuroscientists found in the primary visual cortex (V1 simple cells). Layer 2 combines those edges into parts (curves for digit “8”, straight lines for “1”, loops for “0”). The fully-connected layers integrate everything into a final classification.

What Happens to a Digit “7”

Let's trace what happens when our trained CNN processes a handwritten “7”:

1. Input (1×28×28): The raw pixel values — a bright “7” shape on a dark background.
2. After Conv1 + ReLU (16×28×28): 16 different edge maps. Some highlight the horizontal stroke at the top, others highlight the diagonal stroke going down. Filters that do not match any part of the “7” produce near-zero maps.
3. After Pool1 (16×14×14): Same edge information, but spatially compressed. The exact pixel position of each edge is slightly blurred, but the edges are still clearly present.
4. After Conv2 + ReLU (32×14×14): Higher-level features emerge. One filter might respond to the corner where the horizontal and diagonal strokes meet. Another might respond to the sharp angle at the base.
5. After Pool2 (32×7×7): Compact feature maps encoding the structural properties of the digit.
6. After Flatten + FC layers (10): The logit scores: $[-2.1, -3.4, 0.5, -1.2, -0.8, -2.3, -1.9, \mathbf{9.1}, -0.3, -1.5]$. The score at index 7 is overwhelmingly the highest, so the prediction is digit 7.

Computing the Receptive Field

We just said Conv2 has a 7×7 receptive field. Where does that number come from? The receptive field of a unit at layer $L$ is the patch of input pixels that can influence its value. For a chain of convolutions and pools, there is a clean recurrence (Araujo, Norris & Sim, 2019, Distill):

$r_L = r_{L-1} + (k_L - 1) \cdot \prod_{i=1}^{L-1} s_i$

where $r_L$ is the receptive field at layer $L$, $k_L$ is that layer's kernel size, and the product is the cumulative stride of everything that came before. Start with $r_0 = 1$ (a single input pixel). For our CNN:

So a Conv2 unit actually sees a 8×8 patch of the input (slightly larger than the earlier informal “7×7”; the exact answer depends on whether you count the “Pool1-then-Conv2” path or the “Conv2 directly” path). After Pool2 each unit sees a 10×10 patch — roughly a third of the 28×28 image. That is why Conv2 filters detect curves and corners: they have enough spatial context to compose edges from Conv1 into parts.

The FC Parameter Problem (Bridge to GAP)

Look back at the parameter count for our CNN: FC1 alone contains about 97% of the total parameters. That is a structural problem, not a bug in our code. A fully-connected layer from a 32×7×7 = 1568-dim feature vector to 128 hidden units needs $1568 \times 128 + 128 \approx 200{,}000$ weights, dwarfing the ~5000 parameters in both convolutional layers combined.

Two things follow: (a) most of the model's capacity is spent on a layer that throws away spatial structure, and (b) a bigger input (say 224×224 instead of 28×28) would make FC1 enormous — think tens of millions of parameters for a single FC transition. Early architectures (LeNet, AlexNet, VGG) paid that price. Lin, Chen & Yan (2013) pointed out that a much simpler fix works: Global Average Pooling. Replace Flatten + FC1 with one operation that averages each channel's spatial map down to a single scalar, producing a 32-d vector (one number per channel) with zero parameters.

Why this matters now. Every modern architecture we meet in the next section — AlexNet, VGG, GoogLeNet, ResNet — either uses Global Average Pooling directly (ResNet, GoogLeNet) or is immediately criticised for not using it (VGG's 138 M-parameter fully-connected tail). Keep the FC-parameter problem in mind; it is the thread that ties the historical tour together.

Looking Ahead: In the next section, we will see how the same principles scale to much deeper and more powerful architectures — from LeNet (the original CNN from 1998) through AlexNet, VGG, and the revolutionary ResNet. The building blocks are identical to what we built here. The difference is depth, skip connections, and clever engineering.

References

Every claim about biological vision, architectural history, and regularisation theory in this section is grounded in the papers below. The interactive filter-visualisation and the hierarchy table reproduce findings that the original authors reported; you can read the originals directly.

• Hubel, D. H. & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. Journal of Physiology 160(1), 106–154. DOI: 10.1113/jphysiol.1962.sp006837. — Original discovery of edge-selective “simple cells” in V1.
• LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. (1998). Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE 86(11), 2278–2324. DOI: 10.1109/5.726791. — The LeNet paper; established the conv + pool + FC template we build here.
• Nair, V. & Hinton, G. E. (2010). Rectified Linear Units Improve Restricted Boltzmann Machines. ICML 2010. — Introduced ReLU for deep networks.
• Glorot, X., Bordes, A. & Bengio, Y. (2011). Deep Sparse Rectifier Neural Networks. AISTATS 2011. — Showed ReLU trains faster and deeper than tanh/sigmoid.
• Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. JMLR 15, 1929–1958. — The dropout paper; motivates our p = 0.25.
• Lin, M., Chen, Q. & Yan, S. (2013). Network In Network. ICLR 2014 / arXiv:1312.4400. — Introduced Global Average Pooling as the fix for huge FC layers.
• Araujo, A., Norris, W. & Sim, J. (2019). Computing Receptive Fields of Convolutional Neural Networks. Distill. DOI: 10.23915/distill.00021. — The reference for the recurrence $r_L = r_{L-1} + (k_L - 1) \prod_{i=1}^{L-1} s_i$.