From Pixels to Features: Why CNNs?

Introduction

Convolutional Neural Networks power nearly every modern vision system — from the face-unlock on your phone to the perception stack of a self-driving car. But why convolution specifically? Why not a plain fully-connected network like the one in Chapter 7?

This chapter answers that question in three stages. First, we show why fully-connected networks cannot scale to realistic images. Then we present the three inductive biases — parameter sharing, translation equivariance, and local connectivity — that make convolutions the right tool for pixel data. Finally we build the convolution operation from scratch in 1-D and 2-D so that when you meet nn.Conv2d in the next section, there is nothing mysterious left.

The core claim of this chapter: a convolutional layer is not just “another kind of layer” — it is a fully-connected layer under three strong architectural constraints. Those constraints happen to match the statistics of natural images, and that match is the entire reason CNNs work.

Learning Objectives

After this section you will be able to:

1. Quantify why fully-connected layers fail on images — compute the parameter count for a 224×224×3 input and explain why it is infeasible.
2. Name the three CNN inductive biases — parameter sharing, translation equivariance, and sparse / local connectivity — and explain the statistical assumption each one encodes.
3. Compute 1-D and 2-D convolutions by hand and in NumPy, then reproduce the exact same result with PyTorch's F.conv1d / F.conv2d.
4. Predict output size using $O = \left\lfloor \tfrac{I - K + 2P}{S} \right\rfloor + 1$ for any stride and padding.
5. Explain the feature hierarchy — why stacked conv layers go from edges to textures to object parts, and why this mirrors the mammalian visual cortex
   Reference

   Hubel & Wiesel, 1962, “Receptive fields, binocular interaction and functional architecture in the cat's visual cortex”, J. Physiology 160(1).
   .

The Parameter Explosion Problem

Suppose we want to classify a single RGB image of ImageNet resolution — 224×224 pixels, 3 channels — into 1,000 categories. The most naïve approach is to flatten the image to a vector and push it through a fully-connected (FC) hidden layer.

The numbers, without waving our hands

Flattening the image gives a vector of length $224 \times 224 \times 3 = 150{,}528$. Connecting that to just one hidden neuron requires one weight per input pixel — that is 150,528 weights per neuron. If we want a hidden layer with 1,000 neurons (modest by modern standards), the parameter count is:

$150{,}528 \times 1{,}000 + 1{,}000 \;=\; 150{,}529{,}000 \approx 150\text{M parameters}$

… in a single hidden layer. For comparison, the entire ResNet-50 network (50 layers deep, ImageNet-trained) has about 25M parameters. The fully-connected approach is already six times larger than ResNet-50 after one layer. Training this at scale is impractical for three independent reasons:

1. Memory. 150M float32 weights alone occupy 600 MB — plus gradients, Adam moments, and activations.
2. Sample complexity. Classical learning-theory bounds scale roughly with the parameter count. With this many weights you need an astronomically large training set to avoid overfitting.
3. It ignores structure. Pixel (12, 37) and pixel (12, 38) are almost certainly related — they are neighbours. An FC layer treats them as arbitrary independent features. The spatial topology of the image is thrown away the moment we flatten.

What a 3×3 convolution would use instead

A convolutional layer that produces the same 64 feature channels from a 3-channel input needs only:

$3 \times 3 \times 3 \times 64 + 64 \;=\; 1{,}792 \text{ parameters}$

That is about 84,000× fewer parameters than the FC layer above — and it preserves spatial structure. How is that possible? The interactive diagram below shows the key idea: an FC layer connects every input pixel to every output unit, while a conv layer connects only a tiny local neighbourhood, and shares the same weights across every spatial location.

Three Pillars of CNN Design

A convolutional layer is best understood as a fully-connected layer under three architectural constraints. Each constraint encodes a specific assumption about natural images. If the assumption holds (it does, empirically) we get massive parameter savings and better generalisation for free.

Pillar 1: Parameter sharing

In an FC layer, every output unit has its own private set of input weights. In a conv layer, the same kernel weights are reused at every spatial position. If a 3×3 filter is useful for detecting an edge at pixel (5, 5), it is equally useful at pixel (100, 200) — so we store the filter once and apply it everywhere.

The assumption this encodes is stationarity of image statistics: the distribution of local pixel patterns is roughly the same across the image. That assumption is false for a passport photo centred on a face, but true for natural images on average — and more importantly, the filters we actually want to learn (edges, colour blobs, textures) are themselves translation-independent.

Pillar 2: Translation equivariance

A function $f$ is translation equivariant if shifting the input by $\Delta$ produces an output shifted by the same $\Delta$. Convolution satisfies this exactly:

$(T_{\Delta} I) * K \;=\; T_{\Delta} (I * K)$

Concretely: if a cat appears 50 pixels to the right of where the network saw cats during training, the feature map for “cat” simply shifts 50 pixels — the network does not have to relearn the concept from scratch at every location. This is a structural guarantee of the operation, not something the network has to learn.

Pillar 3: Sparse / local connectivity

Each output unit in a conv layer depends on only a tiny local patch of the input — a $K \times K$ window — rather than every pixel. This matches two well-known facts:

• Image statistics: Nearby pixels are highly correlated; distant pixels are nearly independent. Local filters exploit this correlation; global connections waste capacity on irrelevant pairs.
• Biological precedent: Hubel & Wiesel showed that neurons in the primary visual cortex (V1) respond to small, localised regions of the visual field and to oriented edges within those regions
  Reference

  Hubel & Wiesel, 1962, J. Physiology 160(1). Their single-unit recordings in cat striate cortex were the direct biological inspiration for the Neocognitron (Fukushima 1980), the immediate ancestor of modern CNNs.
  . CNNs are a deliberate computational echo of that architecture.

The whole chapter in one line: a conv layer is an FC layer plus parameter sharing plus locality. The math we are about to do is just the explicit form of those two constraints.

Starting Simple: 1D Convolution

Before we attack images, let us build intuition on a 1-D signal. Exactly the same operation — slide a small filter across the input, sum weighted values — powers audio processing, time-series forecasting, and 1-D sensor fusion.

Intuition: a sliding weighted sum

Imagine a short ruler with three numbers written on it. You lay it on top of a signal, multiply each signal value by the number on the ruler above it, and add the three products. That single number is the output at that position. Slide the ruler one step to the right, repeat.

Mathematical definition

The classical (flipped) definition of 1-D convolution on infinite signals is:

$(f * g)[n] \;=\; \sum_{k=-\infty}^{\infty} f[k]\, g[n - k]$

In deep learning we almost always skip the kernel flip and use the simpler cross-correlation definition, which for finite signals of length $L$ and kernel of length $K$ reads:

$(f \star g)[n] \;=\; \sum_{k=0}^{K-1} f[n + k]\, g[k]$

Worked example in Python — every line, every value

Below is a complete hand-rolled 1-D convolution. Click any line on the right to see what that line does in memory.

1import numpy as np
2
3# Input signal: a simple linearly-increasing ramp
4signal = np.array([1, 2, 3, 4, 5])
5
6# Kernel: the backward-difference filter [1, 0, -1]
7kernel = np.array([1, 0, -1])
8
9# Output has length = len(signal) - len(kernel) + 1 = 5 - 3 + 1 = 3
10output = []
11
12for n in range(len(signal) - len(kernel) + 1):
13    window = signal[n : n + len(kernel)]   # slice 3 consecutive values
14    products = window * kernel             # element-wise multiply (NumPy broadcasting)
15    value = np.sum(products)               # reduce to a single scalar
16    output.append(value)
17
18print("signal:", signal)
19print("kernel:", kernel)
20print("output:", output)
21# Expected → output: [-2, -2, -2]

The same calculation in PyTorch

We now reproduce the identical numerical result using torch.nn.functional.conv1d. The point is to see the mapping between our hand-written loop and the optimised library call — not to use a new algorithm.

1import torch
2import torch.nn.functional as F
3
4# Same signal and kernel as above, but as PyTorch tensors.
5# Conv layers expect shape (batch, channels, length) — we add two dummy axes.
6signal = torch.tensor([1., 2., 3., 4., 5.]).view(1, 1, 5)  # (N=1, C=1, L=5)
7kernel = torch.tensor([1., 0., -1.]).view(1, 1, 3)         # (C_out=1, C_in=1, K=3)
8
9# Cross-correlation (what deep learning calls "convolution")
10output = F.conv1d(signal, kernel, stride=1, padding=0)
11
12print("signal shape :", signal.shape)
13print("kernel shape :", kernel.shape)
14print("output       :", output.squeeze().tolist())
15# Expected → output: [-2.0, -2.0, -2.0]

Interactive 1-D convolution

Drag the kernel across the signal, change the weights, and watch each output value update. This is the same operation we just coded by hand — only now you can feel it move.

2D Convolution for Images

Images are 2-D grids of pixels, so we upgrade the sliding window from a line to a square. Exactly the same principle applies — multiply, sum, slide — just along two axes instead of one.

Mathematical definition

For an image $I$ and a kernel $K$ of size $M \times N$, the (cross-correlation flavour of) 2-D convolution is:

$(I * K)[i, j] \;=\; \sum_{m=0}^{M-1}\sum_{n=0}^{N-1} I[i+m,\, j+n] \cdot K[m, n]$

The intuitive picture

Place a small $3 \times 3$ transparency on the image. Each cell of the transparency carries a number (the kernel weight). At each position:

1. Multiply every pixel under the transparency by its corresponding kernel weight.
2. Add the nine products.
3. Write that single number to the output at this position.
4. Slide the transparency one pixel right (or down), repeat.

Watch it move

The animation below runs exactly the operation we just described. The highlighted 3×3 window on the left is the current position; the output cell it fills in on the right is computed by the multiply-and-sum rule above.

What the kernel weights encode: positive on one side and negative on the other → edge detector. All equal and positive → blur. Large positive centre with negative neighbours → sharpen. In classical vision these were hand-designed; in deep learning they are learned by gradient descent, and the ones that emerge look strikingly like Gabor filters — the same pattern biologists find in V1

Reference

Zeiler & Fergus, 2014, “Visualizing and Understanding Convolutional Networks”, ECCV. Also Olshausen & Field, 1996, Nature 381, on sparse-coding models that produce similar filters from natural image statistics alone.

.

From Edges to Objects: The Feature Hierarchy

A single conv layer can only detect small, local patterns. The reason CNNs work on object recognition is that we stack many of them. Each successive layer sees a slightly larger piece of the input (a growing receptive field, which we formalise in Section 3) and combines the previous layer's features into more complex ones.

This is not a hand-waved analogy. Feature-visualisation techniques applied to trained CNNs — activation maximisation, deconvolutional projections, saliency maps — repeatedly recover the same hierarchy: edges and colour blobs in layer 1, simple textures and corners in layer 2, motifs like eyes and wheels in the mid layers, and full object-level concepts near the top

We now have the what and the why. In Section 2 we zoom in on the convolution operation itself — the distinction between cross-correlation and true convolution, a fully worked 2-D example, interactive kernels, and multi-channel (RGB) convolution. In Section 3 we add stride, padding, pooling, and receptive fields to complete the picture.