Stride, Padding, Pooling & Receptive Fields

Stride — Controlling Output Resolution

In Section 2 our kernel moved one pixel at a time. That is stride 1. If we instead move s pixels per step, we evaluate fewer windows, the output shrinks by roughly a factor of s in each spatial axis, and we save compute proportionally.

Definition

Let $S$ be the stride. Then the output size (no padding) becomes:

$O \;=\; \left\lfloor \frac{I - K}{S} \right\rfloor + 1$

For the 5×5 image and 3×3 Sobel-X kernel of Section 2: stride 1 gives a 3×3 output; stride 2 gives a 2×2 output; stride 3 gives a 1×1 output. Every stride greater than 1 throws information away, but that is often exactly what we want — modern architectures like ResNet use stride-2 convolutions in place of (or alongside) pooling precisely to down-sample

Python & PyTorch — stride in action

[[160. 160. 160.]
 [240. 240. 240.]
 [320. 320. 320.]]

[[160. 160.]
 [320. 320.]]

tensor([[160., 160.],
        [320., 320.]])

1import numpy as np
2import torch
3import torch.nn.functional as F
4
5# Same 5×5 gradient image + Sobel-X kernel as Section 2
6I = np.array([
7    [10, 20, 30, 40, 50],
8    [20, 40, 60, 80, 100],
9    [30, 60, 90, 120, 150],
10    [40, 80, 120, 160, 200],
11    [50, 100, 150, 200, 250],
12], dtype=float)
13
14K = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], dtype=float)
15
16def conv2d_stride(image, kernel, stride=1):
17    H, W = image.shape
18    kH, kW = kernel.shape
19    # Output size depends on stride: O = (I - K) // S + 1
20    out_H = (H - kH) // stride + 1
21    out_W = (W - kW) // stride + 1
22
23    output = np.zeros((out_H, out_W))
24    for i in range(out_H):
25        for j in range(out_W):
26            h0 = i * stride                       # top-left row in the IMAGE
27            w0 = j * stride                       # top-left col in the IMAGE
28            window = image[h0 : h0 + kH, w0 : w0 + kW]
29            output[i, j] = np.sum(window * kernel)
30    return output
31
32out_s1 = conv2d_stride(I, K, stride=1)
33out_s2 = conv2d_stride(I, K, stride=2)
34
35print("stride=1 output shape:", out_s1.shape, "\n", out_s1)
36print("stride=2 output shape:", out_s2.shape, "\n", out_s2)
37
38# PyTorch check — identical numbers
39I_t = torch.tensor(I).unsqueeze(0).unsqueeze(0)
40K_t = torch.tensor(K).unsqueeze(0).unsqueeze(0)
41print("torch stride=2:\n", F.conv2d(I_t, K_t, stride=2).squeeze())
42
43# Expected:
44# stride=1 output: (3, 3)  → [[160,160,160],[240,240,240],[320,320,320]]
45# stride=2 output: (2, 2)  → [[160,160],[320,320]]

Padding — Taming the Boundary

Without padding, every conv layer shrinks the spatial size by $K - 1$. Stack 10 layers with 3×3 kernels and you lose 20 pixels in each dimension. Padding fixes this by adding extra rows and columns around the input, so the kernel has somewhere to land at the boundary.

The four common padding modes

Terminology: ‘valid’, ‘same’, ‘full’

Higher-level libraries often use named padding shortcuts:

• valid — no padding. Output shrinks by $K - 1$.
• same — pad so output spatial size equals input (when stride=1). For odd kernel sizes: $P = (K - 1)/2$.
• full — pad $K - 1$ on each side. Output is larger than input. Rare in deep learning; common in signal processing.

F.pad — all four modes on the same tensor

[[1 2 3]
 [4 5 6]
 [7 8 9]]

[[0 0 0 0 0]
 [0 1 2 3 0]
 [0 4 5 6 0]
 [0 7 8 9 0]
 [0 0 0 0 0]]

[[1 1 2 3 3]
 [1 1 2 3 3]
 [4 4 5 6 6]
 [7 7 8 9 9]
 [7 7 8 9 9]]

[[5 4 5 6 5]
 [2 1 2 3 2]
 [5 4 5 6 5]
 [8 7 8 9 8]
 [5 4 5 6 5]]

[[9 7 8 9 7]
 [3 1 2 3 1]
 [6 4 5 6 4]
 [9 7 8 9 7]
 [3 1 2 3 1]]

1import torch
2import torch.nn.functional as F
3
4# A tiny 3×3 matrix so padding is easy to see
5x = torch.tensor([
6    [1., 2., 3.],
7    [4., 5., 6.],
8    [7., 8., 9.],
9]).unsqueeze(0).unsqueeze(0)   # (1, 1, 3, 3) for F.pad
10
11# pad = (left, right, top, bottom) — 1 pixel each side
12p = (1, 1, 1, 1)
13
14zeros     = F.pad(x, p, mode="constant", value=0)   # zero-padding
15replicate = F.pad(x, p, mode="replicate")            # repeat edge pixel
16reflect   = F.pad(x, p, mode="reflect")              # mirror without duplicating edge
17circular  = F.pad(x, p, mode="circular")             # wrap-around
18
19print("constant (zero):\n", zeros.squeeze())
20print("replicate:\n",      replicate.squeeze())
21print("reflect:\n",        reflect.squeeze())
22print("circular:\n",       circular.squeeze())

The Output-Size Formula

Combining stride and padding, the full output-size formula becomes:

$O \;=\; \left\lfloor \frac{I - K + 2P}{S} \right\rfloor + 1$

This is the single most important piece of arithmetic in CNN design

Pooling — Down-sampling Without Parameters

Stride-2 convolution is one way to halve the spatial resolution; pooling is the other. A pooling layer slides a window across the input and reduces each window to a single number — typically the MAX or the MEAN — with zero learnable parameters. That “no parameters” is not an accident: LeCun's original LeNet-5 used a sub-sampling layer for exactly this reason — it forces some locality-invariance into the representation without adding capacity

Max vs. average pool

Interactive pooling visualiser

The visualiser below lets you toggle between max and average pooling on a 4×4 or 6×6 feature map. Step through each window and confirm the arithmetic matches your mental model.

Pooling by hand — Python and PyTorch

[[1. 3. 2. 4.]
 [5. 6. 1. 2.]
 [7. 2. 3. 1.]
 [4. 8. 5. 6.]]

[[6. 4.]
 [8. 6.]]

[[3.75 2.25]
 [5.25 3.75]]

[[6., 4.],
 [8., 6.]]

[[3.75, 2.25],
 [5.25, 3.75]]

1import numpy as np
2import torch
3import torch.nn.functional as F
4
5# A 4×4 feature map (pretend it came out of a conv layer)
6X = np.array([
7    [1, 3, 2, 4],
8    [5, 6, 1, 2],
9    [7, 2, 3, 1],
10    [4, 8, 5, 6],
11], dtype=float)
12
13def max_pool_2x2(x):
14    H, W = x.shape
15    out = np.zeros((H // 2, W // 2))
16    for i in range(H // 2):
17        for j in range(W // 2):
18            window = x[2*i : 2*i+2, 2*j : 2*j+2]  # 2×2 non-overlapping tile
19            out[i, j] = window.max()
20    return out
21
22def avg_pool_2x2(x):
23    H, W = x.shape
24    out = np.zeros((H // 2, W // 2))
25    for i in range(H // 2):
26        for j in range(W // 2):
27            window = x[2*i : 2*i+2, 2*j : 2*j+2]
28            out[i, j] = window.mean()
29    return out
30
31print("max  :", max_pool_2x2(X).tolist())
32print("avg  :", avg_pool_2x2(X).tolist())
33
34# PyTorch equivalent — zero learnable parameters
35X_t = torch.tensor(X).unsqueeze(0).unsqueeze(0)
36print("torch max:", F.max_pool2d(X_t, kernel_size=2, stride=2).squeeze().tolist())
37print("torch avg:", F.avg_pool2d(X_t, kernel_size=2, stride=2).squeeze().tolist())
38
39# Expected:
40# max: [[6, 4], [8, 6]]
41# avg: [[3.75, 2.25], [5.25, 3.75]]

Receptive Field — Why Depth Matters

Each output unit of a conv layer depends on only a small input neighbourhood. The receptive field of a unit is the set of input pixels that can affect its value. Stacking conv layers grows the receptive field layer by layer — which is the fundamental reason CNN depth helps.

How receptive field grows

For a stack of conv layers with (stride 1, kernel size $K_\ell$) at layer $\ell$, the receptive field after $L$ layers is:

$R_L \;=\; 1 + \sum_{\ell=1}^{L} (K_\ell - 1) \prod_{i=1}^{\ell-1} S_i$

… where $S_i$ is the stride of layer $i$. For an all-stride-1 stack of 3×3 convs the product collapses to 1 and the formula simplifies to $R_L = 1 + 2L$.

Why VGG uses stacks of 3×3: Two 3×3 convs give the same receptive field as one 5×5 conv — but with fewer parameters ($2 \cdot 9 = 18$ vs $25$) and one extra non-linearity in between. Three 3×3 convs match a 7×7 receptive field at a third of the parameters

Reference

Simonyan & Zisserman, 2015, “Very Deep Convolutional Networks for Large-Scale Image Recognition”, ICLR. This is one of the central design lessons of modern CNNs.

.

Interactive: walk through layers, watch the field grow

PyTorch nn.Conv2d Anatomy

Every hyperparameter you have met so far maps directly to an argument of nn.Conv2d:

1import torch
2import torch.nn as nn
3
4conv = nn.Conv2d(
5    in_channels = 3,       # RGB input
6    out_channels = 64,     # 64 filters → 64 output feature maps
7    kernel_size = 3,       # 3×3 spatial window
8    stride = 1,            # move 1 pixel per step
9    padding = 1,           # "same" padding for K=3
10    dilation = 1,          # 1 = ordinary conv; >1 = atrous/dilated
11    groups = 1,            # 1 = full multi-channel; C_in = depthwise
12    bias = True,           # learnable bias per output channel
13)
14
15print(conv.weight.shape)   # torch.Size([64, 3, 3, 3])
16print(conv.bias.shape)     # torch.Size([64])
17
18x = torch.randn(8, 3, 224, 224)   # 8 RGB images, 224×224
19y = conv(x)
20print(y.shape)                    # torch.Size([8, 64, 224, 224])  — same spatial

Manual Implementation (For Reference)

Putting every piece together — stride, padding, multi-channel, bias — into one explicit function. Reading this code is the fastest way to convince yourself that nn.Conv2d holds no mysteries.

1import torch
2import torch.nn.functional as F
3
4def conv2d_manual(x, w, bias=None, stride=1, padding=0):
5    """Multi-channel 2-D cross-correlation — the slow but explicit version."""
6    N, C_in, H, W = x.shape
7    C_out, _, kH, kW = w.shape
8
9    if padding > 0:
10        x = F.pad(x, (padding,) * 4)
11        H += 2 * padding
12        W += 2 * padding
13
14    out_H = (H - kH) // stride + 1
15    out_W = (W - kW) // stride + 1
16    out = torch.zeros(N, C_out, out_H, out_W)
17
18    for n in range(N):
19        for c in range(C_out):
20            for i in range(out_H):
21                for j in range(out_W):
22                    h0, w0 = i * stride, j * stride
23                    window = x[n, :, h0:h0+kH, w0:w0+kW]
24                    out[n, c, i, j] = (window * w[c]).sum()
25
26    if bias is not None:
27        out += bias.view(1, -1, 1, 1)
28    return out
29
30# Verify against PyTorch
31x = torch.randn(2, 3, 8, 8)
32conv = torch.nn.Conv2d(3, 16, kernel_size=3, padding=1)
33ref  = conv(x)
34mine = conv2d_manual(x, conv.weight, conv.bias, stride=1, padding=1)
35print("max abs diff:", (ref - mine).abs().max().item())
36# Expected: ≈ 0 (within float32 round-off)

Putting It All Together: A Full CNN Pipeline

We can now read the full CNN pipeline end to end. Conv layers extract features; pooling (or strided conv) down-samples; the receptive field grows with depth; the final pooled/flattened vector feeds a classifier. The interactive below ties every stage to the concepts of this chapter.

AI / Deep Learning Applications

Every component of this chapter — conv, stride, padding, pooling, receptive field — is used in the production systems below.

Object detection (YOLO, Faster R-CNN)

A CNN backbone extracts features at multiple scales; a head predicts bounding boxes + class probabilities at each spatial location. The increasing receptive field with depth is what lets the network reason about whole objects while still operating on convolutional feature maps

Semantic segmentation (U-Net)

Every pixel gets classified. An encoder of conv + pool layers compresses the image; a decoder of (transposed) conv layers restores spatial resolution; skip connections re-inject fine detail. Medical-image segmentation adopted this architecture wholesale

Neural style transfer

Convolutions factor an image into “content” (responses of deeper layers, roughly object identity) and “style” (Gram-matrix statistics of shallower layers, roughly texture). Optimising an image so its content matches one reference and its style matches another yields Van Gogh-ified photos

Generative image models

StyleGAN, diffusion models, and most modern generators use transposed (fractionally-strided) convolutions to go from low-resolution noise or latent tensors to a full image — the inverse direction of the down-sampling pipeline we have just built

A profound empirical observation

If you visualise the first-layer kernels of a trained ImageNet CNN you see oriented edge detectors, colour blobs, and Gabor-like frequency patterns — strikingly close to what electrophysiology finds in V1 of mammals

Reference

Zeiler & Fergus, 2014, “Visualizing and Understanding Convolutional Networks”, ECCV; Hubel & Wiesel, 1962, J. Physiology 160(1); Olshausen & Field, 1996, Nature 381.

. Nobody trained the network to learn Gabor filters. Gradient descent discovered them from the data.

Summary

Key concepts

Critical formulas

1. 2-D cross-correlation: $(I * K)[i, j] = \sum_m \sum_n I[i+m, j+n] \cdot K[m, n]$
2. Output size: $O = \lfloor (I - K + 2P) / S \rfloor + 1$
3. Parameter count: $K \times K \times C_{\text{in}} \times C_{\text{out}} + C_{\text{out}}$
4. Receptive field for a stride-1, $K$-kernel stack of $L$ layers: $R_L = 1 + L(K - 1)$

Exercises

Conceptual

1. A 128×128×3 image passes through nn.Conv2d(3, 32, kernel_size=5, padding=2, stride=2). What is the output shape and how many parameters does the layer have?
2. Explain in one sentence why Sobel-X, with weights [[-1,0,1],[-2,0,2],[-1,0,1]], detects vertical edges rather than horizontal ones.
3. You need to preserve spatial size with a 7×7 kernel at stride 1. What padding?
4. Two stacked 3×3 convs vs. one 5×5 conv — same receptive field. Give two reasons to prefer the stacked version.
5. Max pool vs. stride-2 conv — when would you reach for each?

Coding

1. Edge magnitude. Apply Sobel-X and Sobel-Y to an image and combine as $\sqrt{G_x^2 + G_y^2}$.
2. Box vs. Gaussian. Apply both to a noisy photograph and explain why Gaussian looks more natural.
3. Verify the formula. Write a test that varies I, K, P, S and asserts F.conv2d output shape matches $\lfloor (I - K + 2P)/S \rfloor + 1$.
4. Implement im2col. Rewrite our manual conv as a single matrix multiplication via unfold. Measure the speedup on a 64×64 input with 128 filters.

References

All factual claims in this chapter are drawn from the following primary sources.

1. Hubel, D. H. and Wiesel, T. N. (1962). “Receptive fields, binocular interaction and functional architecture in the cat's visual cortex.” Journal of Physiology, 160(1), 106–154.
2. Olshausen, B. A. and Field, D. J. (1996). “Emergence of simple-cell receptive field properties by learning a sparse code for natural images.” Nature, 381, 607–609.
3. LeCun, Y., Bottou, L., Bengio, Y. and Haffner, P. (1998). “Gradient-based learning applied to document recognition.” Proceedings of the IEEE, 86(11), 2278–2324.
4. Chellapilla, K., Puri, S. and Simard, P. (2006). “High performance convolutional neural networks for document processing.” Intl. Workshop on Frontiers in Handwriting Recognition.
5. Scherer, D., Müller, A. and Behnke, S. (2010). “Evaluation of pooling operations in convolutional architectures for object recognition.” ICANN.
6. Krizhevsky, A., Sutskever, I. and Hinton, G. E. (2012). “ImageNet classification with deep convolutional neural networks.” NeurIPS.
7. Lin, M., Chen, Q. and Yan, S. (2014). “Network in network.” ICLR.
8. Zeiler, M. D. and Fergus, R. (2014). “Visualizing and understanding convolutional networks.” ECCV.
9. Simonyan, K. and Zisserman, A. (2015). “Very deep convolutional networks for large-scale image recognition.” ICLR.
10. Szegedy, C. et al. (2015). “Going deeper with convolutions” (GoogLeNet / Inception). CVPR.
11. Springenberg, J. T., Dosovitskiy, A., Brox, T. and Riedmiller, M. (2015). “Striving for simplicity: the all convolutional net.” ICLR workshop.
12. Ronneberger, O., Fischer, P. and Brox, T. (2015). “U-Net: convolutional networks for biomedical image segmentation.” MICCAI.
13. He, K., Zhang, X., Ren, S. and Sun, J. (2016). “Deep residual learning for image recognition.” CVPR.
14. Yu, F. and Koltun, V. (2016). “Multi-scale context aggregation by dilated convolutions.” ICLR.
15. Redmon, J., Divvala, S., Girshick, R. and Farhadi, A. (2016). “You only look once: unified, real-time object detection.” CVPR.
16. Luo, W., Li, Y., Urtasun, R. and Zemel, R. (2016). “Understanding the effective receptive field in deep convolutional neural networks.” NeurIPS.
17. Gatys, L. A., Ecker, A. S. and Bethge, M. (2016). “Image style transfer using convolutional neural networks.” CVPR.
18. Radford, A., Metz, L. and Chintala, S. (2016). “Unsupervised representation learning with deep convolutional generative adversarial networks.” ICLR.
19. Goodfellow, I., Bengio, Y. and Courville, A. (2016). Deep Learning. MIT Press. (Chapter 9 — Convolutional Networks — is the standard graduate reference.)
20. Dumoulin, V. and Visin, F. (2016). “A guide to convolution arithmetic for deep learning.” arXiv:1603.07285.
21. Chollet, F. (2017). “Xception: deep learning with depthwise separable convolutions.” CVPR.
22. Howard, A. G. et al. (2017). “MobileNets: efficient convolutional neural networks for mobile vision applications.” arXiv:1704.04861.
23. Ren, S., He, K., Girshick, R. and Sun, J. (2017). “Faster R-CNN: towards real-time object detection with region proposal networks.” IEEE TPAMI, 39(6), 1137–1149.
24. Karras, T., Laine, S. and Aila, T. (2019). “A style-based generator architecture for generative adversarial networks.” CVPR.
25. Ho, J., Jain, A. and Abbeel, P. (2020). “Denoising diffusion probabilistic models.” NeurIPS.

With stride, padding, pooling, and receptive fields now firmly in hand, we can move from individual layers to full CNN architectures. In Chapter 14 we build LeNet-5, VGG, and ResNet from scratch, then use transfer learning to adapt pre-trained models to new tasks.