The Convolution Operation Explained

Cross-Correlation vs Convolution

In signal-processing textbooks, convolution flips the kernel before sliding. In deep-learning libraries it does not. Both operations are called “convolution” — which is a recipe for confusion the first time you read a paper.

The mathematical (flipped) version

True convolution rotates the kernel 180° (i.e. flips it both horizontally and vertically) and then slides:

$(I * K)[i, j] \;=\; \sum_{m}\sum_{n} I[i-m,\, j-n] \cdot K[m, n]$

The flip is what gives convolution the property of commutativity: $f * g = g * f$. It also makes convolution play nicely with the Fourier transform — multiplication in frequency becomes convolution in time/space.

The deep-learning (unflipped) version

Deep-learning frameworks skip the flip. The operation they implement is cross-correlation:

$(I \star K)[i, j] \;=\; \sum_{m}\sum_{n} I[i+m,\, j+n] \cdot K[m, n]$

PyTorch, TensorFlow, JAX, MXNet, Caffe — all of them call this “convolution.” The reason it does not matter in practice is that the kernel weights are learned: whatever flipping we skip, gradient descent absorbs into the weights. A kernel that would detect edges as a cross-correlation can be re-expressed as a (different) kernel that detects the same edges as a true convolution.

Step-by-Step Computation (NumPy)

We will do one 2-D convolution end to end — by hand, then in NumPy, then in PyTorch — so that every number you see in the output is traceable to an explicit pair of window and kernel values.

Input: a 5×5 gradient image

1I = [[10, 20, 30, 40, 50],
2     [20, 40, 60, 80, 100],
3     [30, 60, 90, 120, 150],
4     [40, 80, 120, 160, 200],
5     [50, 100, 150, 200, 250]]

Kernel: 3×3 Sobel-X (vertical-edge detector)

1K = [[-1, 0, 1],
2     [-2, 0, 2],
3     [-1, 0, 1]]

One position by hand: output[0, 0]

The top-left window is the 3×3 block $I[0:3,\, 0:3]$:

1Window I[0:3, 0:3]:     Kernel K:            Element-wise product:
2[[10, 20, 30],          [[-1, 0, 1],         [[-10,   0,  30],
3 [20, 40, 60],     ×     [-2, 0, 2],    =     [-40,   0, 120],
4 [30, 60, 90]]           [-1, 0, 1]]          [-30,   0,  90]]
5
6Sum = -10 + 0 + 30 - 40 + 0 + 120 - 30 + 0 + 90 = 160
7
8→ output[0, 0] = 160

All nine positions — the interactive trace

The code below runs the full loop. Click any line to see what that line does. The loop lines (27, 28, 29, 30) expand into per-iteration cards with the exact window, the element-wise product, and the final sum at each of the 9 output positions. Nothing is hidden.

     c0   c1   c2   c3   c4
r0   10   20   30   40   50
r1   20   40   60   80  100
r2   30   60   90  120  150
r3   40   80  120  160  200
r4   50  100  150  200  250

     c0  c1  c2
r0   -1   0   1
r1   -2   0   2
r2   -1   0   1

[[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]

[[160. 160. 160.]
 [240. 240. 240.]
 [320. 320. 320.]]

[[160. 160. 160.]
 [240. 240. 240.]
 [320. 320. 320.]]

1import numpy as np
2
3# 5×5 input image — a "smooth diagonal gradient"
4I = np.array([
5    [10, 20, 30, 40, 50],
6    [20, 40, 60, 80, 100],
7    [30, 60, 90, 120, 150],
8    [40, 80, 120, 160, 200],
9    [50, 100, 150, 200, 250],
10], dtype=float)
11
12# 3×3 Sobel-X kernel — detects vertical edges (horizontal intensity change)
13K = np.array([
14    [-1, 0, 1],
15    [-2, 0, 2],
16    [-1, 0, 1],
17], dtype=float)
18
19def conv2d_manual(image, kernel):
20    H, W   = image.shape          # input spatial size
21    kH, kW = kernel.shape         # kernel spatial size
22    out_H  = H - kH + 1           # output height (stride=1, padding=0)
23    out_W  = W - kW + 1           # output width
24
25    output = np.zeros((out_H, out_W))
26
27    for i in range(out_H):
28        for j in range(out_W):
29            window = image[i : i + kH, j : j + kW]   # kH × kW patch
30            output[i, j] = np.sum(window * kernel)    # dot product → scalar
31
32    return output
33
34out = conv2d_manual(I, K)
35print("Output shape:", out.shape)
36print(out)
37# Expected (3×3):
38# [[160. 160. 160.]
39#  [240. 240. 240.]
40#  [320. 320. 320.]]

The Same Computation in PyTorch

We now replay the same 2-D convolution with torch.nn.functional.conv2d. Every number should match the NumPy version to the last digit. The point is to see how the hand-rolled loop maps onto a production framework call.

tensor([[160., 160., 160.],
        [240., 240., 240.],
        [320., 320., 320.]])

1import torch
2import torch.nn.functional as F
3
4# Same I and K as above, promoted to PyTorch tensors.
5# F.conv2d expects input (N, C_in, H, W) and weight (C_out, C_in, kH, kW).
6I = torch.tensor([
7    [10., 20., 30., 40., 50.],
8    [20., 40., 60., 80.,100.],
9    [30., 60., 90.,120.,150.],
10    [40., 80.,120.,160.,200.],
11    [50.,100.,150.,200.,250.],
12]).unsqueeze(0).unsqueeze(0)   # shape (1, 1, 5, 5)
13
14K = torch.tensor([
15    [-1., 0., 1.],
16    [-2., 0., 2.],
17    [-1., 0., 1.],
18]).unsqueeze(0).unsqueeze(0)   # shape (1, 1, 3, 3)
19
20out = F.conv2d(I, K, stride=1, padding=0)
21
22print("Input  shape:", I.shape)
23print("Kernel shape:", K.shape)
24print("Output shape:", out.shape)
25print(out.squeeze())
26# Expected → identical to the NumPy version:
27# tensor([[160., 160., 160.],
28#         [240., 240., 240.],
29#         [320., 320., 320.]])

Interactive Convolution Calculator

Click any output cell to see the step-by-step calculation, or press “Animate” to watch the kernel slide across the input. Use the kernel dropdown to see how different weights produce different outputs.

• Identity — output equals input (useful as a sanity check).
• Vertical edge (Sobel X) — high response where brightness changes left → right.
• Horizontal edge (Sobel Y) — high response where brightness changes top → bottom.
• Box blur — smooths by averaging all nine neighbours.
• Sharpen — enhances local differences, hence edges.

Kernel Effects Gallery

Different weights → different outputs, applied to the same input. The gallery below is the whole idea of classical image processing in a box: each filter was hand-designed by someone studying a specific problem.

Why each kernel works

Multi-Channel Convolution

Real images have 3 channels (RGB). Intermediate feature maps can have hundreds. How does convolution handle this? The answer is a single, tidy rule: one kernel spans every input channel and produces exactly one output channel. To get multiple output channels you use multiple kernels.

RGB convolution, spelled out

• Input: shape $(C_{\text{in}} = 3,\, H,\, W)$ — 3 colour channels.
• One kernel: shape $(C_{\text{in}} = 3,\, K,\, K)$ — separate weights for each input channel, covering the same spatial $K \times K$ window.
• Output: shape $(1,\, H',\, W')$ — one scalar per position.

At each output position we compute three element-wise multiply-and-sum operations (one per input channel) and add them together to get one scalar. The diagram below is the visual version of that recipe.

Numerical trace: a 2×2 RGB patch by one 2×2×3 kernel

The code below walks the full arithmetic with tiny integer values so you can check every multiplication in your head.

[[1 2]
 [3 4]]

[[5 6]
 [7 8]]

[[ 9 10]
 [11 12]]

[[1 0]
 [0 1]]

[[0 1]
 [1 0]]

[[1 1]
 [1 1]]

[[1×1, 2×0],   [[1, 0],
 [3×0, 4×1]] =  [0, 4]]

[[5×0, 6×1],   [[0,  6],
 [7×1, 8×0]] =  [7,  0]]

[[ 9×1,10×1],  [[ 9,10],
 [11×1,12×1]]= [11,12]]

1import numpy as np
2
3# Tiny 2×2 RGB image: 3 channels, each 2×2 pixels
4I = np.array([
5    [[1, 2], [3, 4]],      # R channel
6    [[5, 6], [7, 8]],      # G channel
7    [[9,10], [11,12]],     # B channel
8])  # shape (C=3, H=2, W=2)
9
10# One 2×2×3 kernel — same spatial size as the image, so we get a scalar out
11K = np.array([
12    [[1, 0], [0, 1]],      # R weights
13    [[0, 1], [1, 0]],      # G weights
14    [[1, 1], [1, 1]],      # B weights
15])  # shape (C=3, kH=2, kW=2)
16
17# Per-channel contribution — element-wise multiply then sum each channel
18red_contrib   = np.sum(I[0] * K[0])
19green_contrib = np.sum(I[1] * K[1])
20blue_contrib  = np.sum(I[2] * K[2])
21
22# The output pixel is the SUM of the three channel contributions
23output = red_contrib + green_contrib + blue_contrib
24
25print("R contribution:", red_contrib)
26print("G contribution:", green_contrib)
27print("B contribution:", blue_contrib)
28print("output pixel  :", output)
29# Expected → R=5, G=13, B=42, output=60

Multiple output channels

To produce $C_{\text{out}}$ output feature maps we use $C_{\text{out}}$ independent multi-channel kernels. The full weight tensor of a conv layer therefore has shape:

$\mathbf{W} \in \mathbb{R}^{C_{\text{out}} \times C_{\text{in}} \times K \times K}$

The first layer of many CNNs chooses $C_{\text{out}} = 64$: 64 different kernels, each watching all 3 RGB channels. Different kernels converge to detect different features — horizontal edges, vertical edges, diagonals, colour blobs, …

Parameter count

$\text{params} \;=\; K \times K \times C_{\text{in}} \times C_{\text{out}} \;+\; C_{\text{out}}$

… where the trailing $C_{\text{out}}$ is one bias per output channel.

1# A canonical "first conv layer": RGB input, 64 filters, 3×3 kernels
2import torch.nn as nn
3
4conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3)
5params = sum(p.numel() for p in conv1.parameters())
6print(f"Parameters: {params}")
7# 3 × 3 × 3 × 64 + 64 = 1,728 + 64 = 1,792

We now have the operation itself down cold — in math, in NumPy, and in PyTorch. Section 3 adds the three knobs that turn this raw operation into a practical CNN building block: stride, padding, and pooling. It also introduces the concept that explains why CNN depth matters — the receptive field.