Motivation for Convolutions

Introduction

We've spent the previous chapters building fully connected neural networks that can learn complex functions. These networks are universal function approximators—given enough neurons, they can theoretically learn any function. So why do we need a new architecture for images?

The short answer: images are special. They have structure that fully connected networks ignore, and exploiting this structure leads to dramatically better models. Convolutional Neural Networks (CNNs) were designed specifically to leverage the spatial structure inherent in visual data.

The Core Insight: Fully connected networks treat every input pixel as independent, ignoring the fact that nearby pixels are highly correlated and that the same patterns (edges, textures, shapes) appear throughout an image.

In this section, we'll build the motivation for convolutions from first principles. By the end, you'll understand why convolutions are not just an optimization trick, but a fundamentally better way to process visual information.

The Fully Connected Problem

What Happens When We Apply MLPs to Images?

Let's think about what happens when we try to classify images using a standard fully connected network. Consider a modest $224 \times 224$ RGB image—a common input size for image classification.

Counting Parameters

First, we need to flatten the image into a vector:

Now, if our first hidden layer has just 1,000 neurons (quite modest), we need:

That's over 150 million parameters in just the first layer! For comparison, the entire VGG-16 network (a large CNN) has about 138 million parameters total.

Problems with the Fully Connected Approach

1. Memory and Computation: Storing and computing with billions of parameters is prohibitively expensive
2. Overfitting: With so many parameters and relatively few training examples, the network will memorize rather than generalize
3. Wasted Capacity: Most of these parameters learn nothing useful because they connect unrelated pixels
4. No Spatial Awareness: The network treats pixel (0,0) and pixel (223,223) as equally related to pixel (112,112)

224 * 224 * 3 = 150,528

fc_layer.weight.shape = (1000, 150528)

3 × 64 × 3 × 3 + 64 = 1,792

1import torch.nn as nn
2
3# Parameters for fully connected network on ImageNet-sized input
4input_size = 224 * 224 * 3  # 150,528
5hidden_size = 1000
6
7fc_layer = nn.Linear(input_size, hidden_size)
8print(f"Parameters: {sum(p.numel() for p in fc_layer.parameters()):,}")
9# Output: Parameters: 150,529,000
10
11# For comparison: a convolutional layer
12conv_layer = nn.Conv2d(3, 64, kernel_size=3, padding=1)
13print(f"Conv parameters: {sum(p.numel() for p in conv_layer.parameters()):,}")
14# Output: Conv parameters: 1,792
15
16# Ratio: 150,529,000 / 1,792 = 84,000x more parameters!

The Curse of Dimensionality

The parameter explosion is a symptom of a deeper problem: the curse of dimensionality. As input dimensionality grows, the amount of data needed to adequately cover the space grows exponentially.

Why More Parameters Means More Data

A rough rule of thumb in machine learning: you need at least 5-10 training examples per parameter to avoid severe overfitting. Let's apply this to our fully connected image classifier:

The Sparsity of Image Space

Here's another way to think about the problem. A $28 \times 28$ grayscale image has $256^{784}$ possible configurations. That's approximately $10^{1888}$ possible images—more than the number of atoms in the observable universe ($\sim 10^{80}$).

Yet most of these configurations are meaningless noise. Real images occupy a tiny, highly structured subspace. A good model should encode this structure, not try to memorize every possible pixel configuration.

Images Have Spatial Structure

The key insight that motivates convolutions: images are not random collections of pixels. They have rich spatial structure that we can exploit.

Property 1: Locality

Nearby pixels are more related than distant pixels.

When you look at a pixel in an image, its immediate neighbors tell you almost everything about it. The pixel at location (100, 100) is highly correlated with pixels at (99, 100), (101, 100), (100, 99), and (100, 101). It has almost no correlation with the pixel at (0, 0).

This suggests: don't connect every input to every output. Instead, connect each output to only a local neighborhood of inputs.

1import numpy as np
2from scipy.ndimage import correlate
3
4# Load any image (simulated here)
5np.random.seed(42)
6image = np.random.randn(100, 100)
7
8# Correlation between adjacent pixels
9adjacent_corr = np.corrcoef(image[:-1, :].flatten(),
10                            image[1:, :].flatten())[0, 1]
11print(f"Adjacent pixel correlation: {adjacent_corr:.4f}")
12# For natural images: typically 0.9+
13
14# Correlation between distant pixels
15distant_corr = np.corrcoef(image[:50, :50].flatten(),
16                           image[50:, 50:].flatten())[0, 1]
17print(f"Distant pixel correlation: {distant_corr:.4f}")
18# For natural images: typically near 0

Property 2: Translation Invariance

The same patterns appear at different locations.

A cat's ear looks like a cat's ear whether it appears in the top-left corner, center, or bottom-right of the image. An edge is an edge regardless of where it is. This means the same feature detector should be useful everywhere in the image.

Implication: We should use the same weights (parameters) at every spatial location. This is called parameter sharing or weight tying.

Property 3: Compositionality

Complex patterns are built from simpler patterns.

A face is made of eyes, nose, and mouth. An eye is made of edges forming specific shapes. This hierarchical structure suggests that we should build layers of feature detectors, where each layer combines features from the previous layer.

Three Key Insights That Define Convolutions

Based on the properties of images, we can identify three key design principles that convolutions implement:

1. Sparse Connectivity (Local Receptive Fields)

Instead of connecting each output to all inputs, connect it to only a small, local region called the receptive field.

This reduces parameters by a factor of almost 100,000x!

2. Parameter Sharing (Weight Tying)

Use the same weights at every spatial location. A $3 \times 3$ edge detector applied to the top-left uses the same 9 weights as when applied to the bottom-right.

1# In a fully connected layer, every connection has unique weights
2# fc_layer[i, j] != fc_layer[i, k] for different input positions j, k
3
4# In a convolutional layer, the same kernel is applied everywhere
5kernel = torch.tensor([
6    [-1, 0, 1],
7    [-2, 0, 2],
8    [-1, 0, 1]
9], dtype=torch.float32)  # Sobel edge detector
10
11# This SAME kernel is applied at position (0,0), (0,1), (1,0), ...
12# The kernel "slides" across the entire image

3. Translation Equivariance

Because we use the same weights everywhere, if the input shifts, the output shifts by the same amount. This property is called translation equivariance.

If a cat moves from the left to the right of the image, the feature maps detecting "cat parts" also move correspondingly.

Interactive: Convolution in Action

Now that we understand the theory, let's see convolution in action. The interactive demos below let you explore how convolutions work step by step.

Watch the Kernel Slide

The convolution operation slides a small kernel (filter) across the input image, computing a weighted sum at each position. Press play to watch it happen, or step through manually.

Receptive Field Growth

As we stack more convolutional layers, each output neuron "sees" a larger region of the original input. This is called the receptive field. Click through the layers to see how it grows.

Biological Inspiration

The design of CNNs was heavily inspired by neuroscience, particularly the work of Hubel and Wiesel on the visual cortex (Nobel Prize, 1981).

The Visual Cortex

Key discoveries about how mammals process visual information:

1. Simple Cells: Neurons that respond to edges at specific orientations in specific locations. Like our convolutional filters!
2. Complex Cells: Neurons that respond to edges regardless of exact position within their receptive field. Like our pooling layers!
3. Hierarchical Processing: Visual information flows through layers V1 → V2 → V4 → IT, with increasingly abstract representations. Like our deep CNN layers!

Edge Detection: A Motivating Example

Before diving into the mathematics of convolution, let's see a concrete example of why local operations are powerful. Edge detection is one of the most fundamental operations in image processing.

The Sobel Filter

The Sobel filter detects edges by computing the gradient (rate of change) of pixel intensities:

At an edge: -1(0) + 1(1) + -2(0) + 2(1) + -1(0) + 1(1) = 4

edges_x.shape = (1, 1, 32, 32)

magnitude = √(Gx² + Gy²)

1import torch
2import torch.nn.functional as F
3import matplotlib.pyplot as plt
4
5# Sobel filters for horizontal and vertical edges
6sobel_x = torch.tensor([
7    [-1, 0, 1],
8    [-2, 0, 2],
9    [-1, 0, 1]
10], dtype=torch.float32).view(1, 1, 3, 3)
11
12sobel_y = torch.tensor([
13    [-1, -2, -1],
14    [ 0,  0,  0],
15    [ 1,  2,  1]
16], dtype=torch.float32).view(1, 1, 3, 3)
17
18# Create a simple test image with a vertical edge
19image = torch.zeros(1, 1, 32, 32)
20image[:, :, :, 16:] = 1.0  # Right half is white
21
22# Apply Sobel filters (this IS convolution!)
23edges_x = F.conv2d(image, sobel_x, padding=1)
24edges_y = F.conv2d(image, sobel_y, padding=1)
25
26# Combine for edge magnitude
27edges = torch.sqrt(edges_x**2 + edges_y**2)
28
29print(f"Input shape: {image.shape}")
30print(f"Output shape: {edges.shape}")
31print(f"Max edge response: {edges.max():.2f}")

What the Sobel Filter Computes

At each pixel location, the Sobel filter computes a weighted sum of the $3 \times 3$ neighborhood:

This computes the horizontal gradient: bright pixels on the right minus bright pixels on the left. Large values indicate a vertical edge.

Why This Matters

• Only 9 parameters: The entire operation uses just 9 numbers, not millions
• Works everywhere: The same 9 weights detect edges anywhere in the image
• Meaningful output: The result tells us something useful about image structure
• Composable: We can stack edge detectors to find corners, then shapes, then objects

From Fully Connected to Convolution

Let's see how convolution relates to fully connected layers, and why it's a strict generalization.

A Fully Connected Layer as a Sparse Matrix

Consider a tiny $4 \times 4$ image flattened to 16 pixels, connected to a $4 \times 4$ output (also 16 values). A fully connected layer has a $16 \times 16 = 256$ weight matrix where every input connects to every output.

A convolutional layer with a $3 \times 3$ kernel can be viewed as the same $16 \times 16$ matrix, but with two constraints:

1. Sparsity: Most entries are zero (only local connections exist)
2. Weight Sharing: Non-zero entries share the same values

nn.Conv2d(in=1, out=1, kernel=2)

kernel = [[1, 2], [3, 4]]

[[0,1,2,3], [4,5,6,7], [8,9,10,11], [12,13,14,15]]

1(0) + 2(1) + 3(4) + 4(5) = 34

1import torch
2import torch.nn as nn
3
4# Create a simple 2x2 convolution on a 4x4 input
5conv = nn.Conv2d(1, 1, kernel_size=2, bias=False)
6conv.weight.data = torch.tensor([[[[1, 2], [3, 4]]]], dtype=torch.float32)
7
8# Input: 4x4 image
9x = torch.arange(16, dtype=torch.float32).view(1, 1, 4, 4)
10print("Input:")
11print(x.squeeze())
12
13# Apply convolution
14y = conv(x)
15print("Output (via conv):")
16print(y.squeeze())
17
18# Verify: manual computation for position (0,0)
19# output[0,0] = 1*input[0,0] + 2*input[0,1] + 3*input[1,0] + 4*input[1,1]
20manual = 1*0 + 2*1 + 3*4 + 4*5
21print(f"Manual computation for (0,0): {manual}")  # Matches y[0,0]

The Efficiency Gain

For a $224 \times 224$ image with a $3 \times 3$ convolution:

What We Give Up

The constraints of convolution mean we cannot learn certain functions:

• Global dependencies: A single conv layer cannot relate pixels far apart
• Position-specific processing: We cannot learn "if pixel is in top-left, do X; if bottom-right, do Y"
• Asymmetric relationships: The relationship between positions (0,0) and (1,0) must be the same as between (50,50) and (51,50)

For images, these constraints are features, not bugs! They encode the prior knowledge that images have translation-invariant local structure.

Real-World Applications

CNNs have revolutionized computer vision and beyond. Here are some real-world applications that demonstrate the power of convolutional architectures:

Industry Impact

• Healthcare: CNNs detect diabetic retinopathy, skin cancer, and COVID-19 from X-rays with accuracy matching or exceeding human experts
• Agriculture: Drone-mounted CNN systems identify crop diseases, estimate yields, and optimize irrigation
• Manufacturing: Visual inspection systems detect defects in products at superhuman speed and accuracy
• Security: Surveillance systems use CNNs for face recognition, anomaly detection, and threat identification
• Entertainment: Video games and movies use CNNs for real-time style transfer, upscaling, and visual effects

Summary

We've established why convolutions are essential for processing images:

Key Takeaways

1. Fully connected layers have too many parameters for high-dimensional inputs like images, leading to overfitting and computational issues
2. Images have spatial structure: locality (nearby pixels are related), translation invariance (patterns repeat), and compositionality (complex features built from simple ones)
3. Convolutions exploit this structure through sparse connectivity, parameter sharing, and translation equivariance
4. The design is biologically inspired by how the visual cortex processes information
5. Convolution is a constrained fully connected layer—we trade expressiveness for efficiency and better inductive bias

Exercises

Conceptual Questions

1. Calculate the number of parameters in the first layer of a fully connected network that takes a $64 \times 64 \times 3$ image and outputs 512 features. Compare to a $5 \times 5$ convolution with 64 output channels.
2. Why would a fully connected network struggle to classify an image of a cat in the top-left corner if it was only trained on cats in the center?
3. Explain the difference between translation equivariance and translation invariance. Which does convolution provide, and how do we achieve the other?
4. In what scenarios would you NOT want to use convolutions? Give at least two examples.

Coding Exercises

1. Implement a vertical edge detector and horizontal edge detector as $3 \times 3$ convolutions. Apply them to a checkerboard image and visualize the results.
2. Write code to demonstrate translation equivariance: shift an input image by 5 pixels, apply a convolution, and show that the output is also shifted by 5 pixels.
3. Create a fully connected layer that implements a $3 \times 3$ convolution on a $6 \times 6$ input (hint: most weights will be zero, and many will be shared).

Challenge Exercise

Implement a CNN from scratch using only NumPy. Create a simple 2-layer CNN (conv → relu → conv → relu → flatten → fc) and train it on a toy dataset like a simplified MNIST (e.g., just digits 0 and 1, resized to 14×14). This will solidify your understanding of forward and backward passes through convolutional layers.

In the next section, we'll dive deep into the mathematics of the convolution operation itself, understanding exactly how kernels slide across images to produce feature maps.