Convolution Parameters

Introduction

In Section 10.1 we argued why convolutions exist; in Section 10.2 we defined what a convolution is and worked out a single $5 \times 5$ example by hand. We now answer the natural follow-up: how do we control the shape, receptive field, and computational cost of a convolutional layer?

Three knobs do essentially all the work:

• Stride $S$ — how far the kernel jumps between evaluations. Controls output resolution.
• Padding $P$ — how we treat the boundary. Controls whether spatial size shrinks each layer.
• Dilation $D$ — how far apart the kernel's taps sit. Controls the receptive field for a fixed parameter budget.

The one sentence summary: stride shrinks the output, padding prevents that shrinkage, and dilation enlarges the field of view without adding parameters. Master these three and you can read any CNN architecture diagram on sight.

A single formula from Dumoulin & Visin (2016) [1] ties them all together, and it is exactly the formula PyTorch uses internally for nn.Conv2d [2]. We will derive it, implement it twice (NumPy first, then PyTorch), and verify every number produced along the way.

Learning Objectives

1. Predict output shape for any combination of $I, K, P, S, D$ using the master formula, and know when you need the floor function.
2. Choose stride vs. pooling appropriately for downsampling — including why strided convolutions replaced max-pooling in many modern architectures (Springenberg et al., 2015) [3].
3. Pick a padding mode (zeros, reflect, replicate, circular) based on what the input actually represents.
4. Reason about dilation and compute the receptive field of a stack of dilated convolutions — the core idea behind WaveNet [4] and DeepLab [5].
5. Implement each knob twice: once from scratch in NumPy, once with PyTorch, and verify that both produce the same numbers.

The Master Output-Shape Formula

For a 1-D spatial dimension with input size $I$, kernel size $K$, padding $P$ applied on each side, stride $S$, and dilation $D$, the output size is

$\; O \;=\; \left\lfloor \dfrac{I + 2P - D(K-1) - 1}{S} \right\rfloor + 1 \;$

This is the exact formula listed in the PyTorch documentation for nn.Conv2d [2] and derived geometrically in Dumoulin & Visin [1]. It applies independently to height and width (2-D images have two such equations).

Breaking It Down

Stride — Controlling Output Resolution

Stride $S$ is how many pixels the kernel jumps between evaluations. $S = 1$ is dense sampling; every pixel gets its own output. $S = 2$ visits every other pixel along each axis, so the output footprint is roughly a quarter of the input.

Intuition

Imagine the kernel as a flashlight sweeping a wall. Stride is the distance between successive flashes. A fast sweep (large stride) covers the whole wall in fewer frames but loses detail; a slow sweep (small stride) gives you more frames of fine-grained information at higher cost.

Plugging $P = 0, D = 1$ into the master formula gives the stride-only special case $O = \lfloor (I - K)/S \rfloor + 1$.

Stride in Pure Python

Let's start where every working implementation of a new idea should: with a few lines of NumPy that we can trace on paper. The input is the same $5 \times 5$ grid we used in Section 10.2, and we compare stride = 1 against stride = 2.

      c0   c1   c2   c3   c4
r0   1    2    3    4    5
r1   6    7    8    9   10
r2  11   12   13   14   15
r3  16   17   18   19   20
r4  21   22   23   24   25

     c0  c1  c2
r0    1   0   1
r1    0   1   0
r2    1   0   1

img[1:4, 1:4] →
  7   8   9
 12  13  14
 17  18  19

img[2:5, 2:5] →
 13  14  15
 18  19  20
 23  24  25

     c0  c1  c2
r0  35  40  45
r1  60  65  70
r2  85  90  95

     c0  c1
r0  35  45
r1  85  95

1import numpy as np
2
3I = np.array([
4    [1,  2,  3,  4,  5],
5    [6,  7,  8,  9, 10],
6    [11, 12, 13, 14, 15],
7    [16, 17, 18, 19, 20],
8    [21, 22, 23, 24, 25],
9], dtype=float)
10
11K = np.array([
12    [1, 0, 1],
13    [0, 1, 0],
14    [1, 0, 1],
15], dtype=float)
16
17def conv2d_strided(img, kern, stride):
18    H, W = img.shape
19    k, _ = kern.shape
20    out_H = (H - k) // stride + 1
21    out_W = (W - k) // stride + 1
22    out = np.zeros((out_H, out_W))
23    for i in range(out_H):
24        for j in range(out_W):
25            window = img[i*stride : i*stride + k,
26                         j*stride : j*stride + k]
27            out[i, j] = np.sum(window * kern)
28    return out
29
30print("stride=1 →")
31print(conv2d_strided(I, K, stride=1))
32# [[35. 40. 45.]
33#  [60. 65. 70.]
34#  [85. 90. 95.]]
35
36print("stride=2 →")
37print(conv2d_strided(I, K, stride=2))
38# [[35. 45.]
39#  [85. 95.]]

Stride in PyTorch

PyTorch expresses stride through the stride keyword of both F.conv2d and nn.Conv2d. The math is identical; only the tensor layout (NCHW) and the call site differ.

[[35, 40, 45],
 [60, 65, 70],
 [85, 90, 95]]

[[35, 45],
 [85, 95]]

1import torch
2import torch.nn.functional as F
3
4# Same 5x5 input and 3x3 kernel, but in PyTorch's NCHW layout
5I = torch.tensor([
6    [1,  2,  3,  4,  5],
7    [6,  7,  8,  9, 10],
8    [11, 12, 13, 14, 15],
9    [16, 17, 18, 19, 20],
10    [21, 22, 23, 24, 25],
11], dtype=torch.float32).view(1, 1, 5, 5)
12
13K = torch.tensor([
14    [1, 0, 1],
15    [0, 1, 0],
16    [1, 0, 1],
17], dtype=torch.float32).view(1, 1, 3, 3)
18
19# Functional API — lets us pass weights explicitly
20out_s1 = F.conv2d(I, K, stride=1)
21out_s2 = F.conv2d(I, K, stride=2)
22
23print("stride=1 shape:", out_s1.shape)   # torch.Size([1, 1, 3, 3])
24print(out_s1.squeeze())
25# tensor([[35., 40., 45.],
26#         [60., 65., 70.],
27#         [85., 90., 95.]])
28
29print("stride=2 shape:", out_s2.shape)   # torch.Size([1, 1, 2, 2])
30print(out_s2.squeeze())
31# tensor([[35., 45.],
32#         [85., 95.]])

Padding — Handling the Boundary

Without padding, every convolution with $K > 1$ shrinks the output. After n stride-1 layers with $K = 3$, a$32 \times 32$ input becomes $(32 - 2n) \times (32 - 2n)$; by 16 layers it has vanished. Padding prevents that collapse by inserting extra rows and columns around the border before the convolution runs.

Valid, Same, and Full Padding

“Same” is really the recipe $P = (k_\text{eff} - 1)/2$, which reduces to $(K-1)/2$ when $D = 1$ and $K$ is odd. Even $K$ requires asymmetric padding, which is why library support is rare.

Padding Modes (Zero, Reflect, Replicate, Circular)

PyTorch's nn.Conv2d supports four padding_modes [2], each appropriate for different data. Toggle the visualizer below to see the effect of each choice on a 5×5 input with $P = 2$:

Padding in Pure Python

With NumPy's np.pad we can implement all four modes without leaving the standard library. Note how padding becomes a pre-processing step: we enlarge the input, then run the same unpadded convolution we already wrote.

[[1,0,1],[0,1,0],[1,0,1]]

[[ 0,  0,  0,  0,  0,  0,  0],
 [ 0,  1,  2,  3,  4,  5,  0],
 [ 0,  6,  7,  8,  9, 10,  0],
 [ 0, 11, 12, 13, 14, 15,  0],
 [ 0, 16, 17, 18, 19, 20,  0],
 [ 0, 21, 22, 23, 24, 25,  0],
 [ 0,  0,  0,  0,  0,  0,  0]]

[[ 8, 16, 19, 22, 14],
 [20, 35, 40, 45, 28],
 [35, 60, 65, 70, 43],
 [50, 85, 90, 95, 58],
 [38, 56, 59, 62, 44]]

1import numpy as np
2
3I = np.array([
4    [1,  2,  3,  4,  5],
5    [6,  7,  8,  9, 10],
6    [11, 12, 13, 14, 15],
7    [16, 17, 18, 19, 20],
8    [21, 22, 23, 24, 25],
9], dtype=float)
10
11K = np.array([
12    [1, 0, 1],
13    [0, 1, 0],
14    [1, 0, 1],
15], dtype=float)
16
17def conv2d_padded(img, kern, padding=0, pad_mode='constant'):
18    # Pad the image first, then do a stride-1 conv
19    if padding > 0:
20        img = np.pad(img, padding, mode=pad_mode)
21    H, W = img.shape
22    k, _ = kern.shape
23    out_H = H - k + 1
24    out_W = W - k + 1
25    out = np.zeros((out_H, out_W))
26    for i in range(out_H):
27        for j in range(out_W):
28            out[i, j] = np.sum(img[i:i+k, j:j+k] * kern)
29    return out
30
31# Same padding: p = (k-1)//2 keeps output = input
32valid_out = conv2d_padded(I, K, padding=0)
33same_out  = conv2d_padded(I, K, padding=1, pad_mode='constant')
34
35print("valid:", valid_out.shape)   # (3, 3)
36print("same :", same_out.shape)    # (5, 5)
37print(same_out)
38# [[ 8. 16. 19. 22. 14.]
39#  [20. 35. 40. 45. 28.]
40#  [35. 60. 65. 70. 43.]
41#  [50. 85. 90. 95. 58.]
42#  [38. 56. 59. 62. 44.]]

Padding in PyTorch

In PyTorch, padding is a keyword on both the functional and module APIs. The string shortcut padding='same' is convenient but only legal when every spatial stride is 1 — an easy-to-miss restriction.

1import torch
2import torch.nn as nn
3import torch.nn.functional as F
4
5I = torch.arange(1, 26, dtype=torch.float32).view(1, 1, 5, 5)
6K = torch.tensor([[1, 0, 1],
7                  [0, 1, 0],
8                  [1, 0, 1]], dtype=torch.float32).view(1, 1, 3, 3)
9
10# 1) Functional API — pass padding explicitly
11valid = F.conv2d(I, K, padding=0)          # "valid"
12same  = F.conv2d(I, K, padding=1)          # "same" when stride=1, k=3
13
14print("valid:", valid.shape)  # (1, 1, 3, 3)
15print("same :", same.shape)   # (1, 1, 5, 5)
16
17# 2) Module API — Conv2d stores weights and padding config
18conv_same = nn.Conv2d(
19    in_channels=1, out_channels=1,
20    kernel_size=3,
21    padding=1,
22    padding_mode='reflect',   # 'zeros' | 'reflect' | 'replicate' | 'circular'
23    bias=False,
24)
25
26# Replace the random init with our fixed kernel to see deterministic output
27with torch.no_grad():
28    conv_same.weight.copy_(K)
29
30y = conv_same(I)
31print(y.shape)                 # torch.Size([1, 1, 5, 5])
32
33# 3) String shortcut (PyTorch >= 1.9)
34conv_auto = nn.Conv2d(1, 1, kernel_size=3, padding='same', bias=False)
35# padding='same' auto-computes p so that H_out == H_in (stride=1 only)

Dilation — Larger Receptive Field for Free

Dilation (also called “atrous” convolution, from the French à trous = “with holes”) inserts $D - 1$ zeros between adjacent kernel taps. The result behaves like a much larger kernel for the purpose of receptive field, while still storing only the original $K \times K$ weights.

The formalism is due to Yu & Koltun (ICLR 2016) [8], who used it to capture multi-scale context in dense prediction. WaveNet [4] uses stacks of dilated causal 1-D convolutions with dilations $1, 2, 4, 8, \ldots$ to model seconds of audio with only a handful of layers. DeepLab [5] uses them (“atrous spatial pyramid pooling”) to capture multi-scale image context for semantic segmentation.

Visualizing Dilation

Drag the slider to see how the effective kernel size $k_\text{eff} = D(K-1) + 1$ grows while the parameter count stays fixed at 9:

Dilation in Pure Python

Implementing dilation is a one-line change to the Python loop: where we used to index img[i+a, j+b], we now index img[i+a·D, j+b·D]. The formula for the output size uses $k_\text{eff}$ in place of $K$.

[[65.]]

1import numpy as np
2
3I = np.arange(1, 26, dtype=float).reshape(5, 5)
4K = np.array([
5    [1, 0, 1],
6    [0, 1, 0],
7    [1, 0, 1],
8], dtype=float)
9
10def conv2d_dilated(img, kern, dilation=1):
11    H, W = img.shape
12    k, _ = kern.shape
13    k_eff = dilation * (k - 1) + 1       # effective kernel size
14    out_H = H - k_eff + 1
15    out_W = W - k_eff + 1
16    out = np.zeros((out_H, out_W))
17    for i in range(out_H):
18        for j in range(out_W):
19            s = 0.0
20            for a in range(k):
21                for b in range(k):
22                    s += img[i + a * dilation,
23                             j + b * dilation] * kern[a, b]
24            out[i, j] = s
25    return out
26
27print("D=1 →", conv2d_dilated(I, K, dilation=1).shape)
28# (3, 3)
29
30print("D=2 →", conv2d_dilated(I, K, dilation=2))
31# [[65.]]
32# The sole output samples the 5 non-zero taps at (0,0),(0,4),(2,2),(4,0),(4,4)
33# → 1 + 5 + 13 + 21 + 25 = 65

Dilation in PyTorch

PyTorch exposes dilation as a keyword on both F.conv2d and nn.Conv2d. The “same” padding recipe generalises cleanly: $P = D(K-1)/2$ for odd $K$.

1import torch
2import torch.nn.functional as F
3
4I = torch.arange(1, 26, dtype=torch.float32).view(1, 1, 5, 5)
5K = torch.tensor([[1, 0, 1],
6                  [0, 1, 0],
7                  [1, 0, 1]], dtype=torch.float32).view(1, 1, 3, 3)
8
9# Dilation = 1 → ordinary conv
10y1 = F.conv2d(I, K, dilation=1)
11print("D=1 shape:", y1.shape)         # (1, 1, 3, 3)
12
13# Dilation = 2 → 5x5 effective kernel on 5x5 input → 1x1 output
14y2 = F.conv2d(I, K, dilation=2)
15print("D=2 shape:", y2.shape)         # (1, 1, 1, 1)
16print("D=2 value:", y2.item())        # 65.0
17
18# Dilation = 2 WITH padding to keep spatial size
19#   k_eff = 2*(3-1)+1 = 5, so p = (5-1)/2 = 2
20y2_same = F.conv2d(I, K, dilation=2, padding=2)
21print("D=2 same:", y2_same.shape)     # (1, 1, 5, 5)

Interactive Parameter Playground

Now that each knob is understood in isolation, try combining them. The playground below runs a live convolution with a fixed 3×3 kernel on a fixed 5×5 input; move the stride, padding, and dilation sliders to see the effective kernel, the output shape, and the numeric output update in real time:

Receptive Field Arithmetic

The receptive field (RF) of a unit in a deep feature map is the size of the region in the input image that can influence its value. Stride, dilation, and depth all increase it. Araujo et al. (2019) [9] derive the following recursion — this is the formula you should commit to memory:

$r_\ell \;=\; r_{\ell - 1} + (k_\ell - 1)\, d_\ell \prod_{i=1}^{\ell - 1} s_i, \qquad r_0 = 1$

In words: each new layer adds $(k-1) d$ “input pixels” to the RF, scaled by the product of all previous strides (the cumulative downsampling, called the jump). Pooling layers obey the same formula — they just use different $k, s$ values.

Worked Example: three 3×3 convs, stride 1

With $k = 3, s = 1, d = 1$ at every layer: $r_0 = 1$, $r_1 = 1 + 2 = 3$, $r_2 = 3 + 2 = 5$, $r_3 = 5 + 2 = 7$. Three 3×3 convs “see” a 7×7 window — which is why VGG [10] stacks two 3×3s instead of using a single 5×5: same receptive field, fewer parameters (18 vs 25 per channel pair), more non-linearities.

Worked Example: WaveNet-style dilated stack

Four 1-D causal convolutions with $k = 2, s = 1$ and dilations $1, 2, 4, 8$ give: $r_1 = 1 + 1 \cdot 1 = 2$, $r_2 = 2 + 1 \cdot 2 = 4$, $r_3 = 4 + 1 \cdot 4 = 8$, $r_4 = 8 + 1 \cdot 8 = 16$. The receptive field doubles every layer. Stacking such blocks is how WaveNet [4] reaches audio context windows of thousands of samples with fewer than 20 layers.

Common Design Patterns

The VGG pattern: stack of same-padded 3×3s

$K = 3, S = 1, P = 1, D = 1$ everywhere, punctuated by stride-2 or pool-2 every few layers. Simonyan & Zisserman [10] showed that this beats larger kernels because two stacked 3×3 layers have the same 5×5 receptive field but use $2 \cdot 3^2 C_\text{in} C_\text{out} = 18 C_\text{in} C_\text{out}$ parameters instead of $25 C_\text{in} C_\text{out}$, and they insert an extra non-linearity.

The ResNet pattern: stride-2 conv for downsampling

He et al. [6] downsample with $K = 3, S = 2, P = 1$ (halves resolution, still “same-ish” at stride-2) and rely on 1×1 convs inside the bottleneck to change channel counts cheaply. The pool-free downsampling preserves gradient flow through residual connections.

The WaveNet pattern: exponentially dilated causal convolutions

Dilation schedule $1, 2, 4, \ldots, 2^{L-1}$ with $K = 2, S = 1$ yields $r_L = 2^L$. L=10 layers already reach 1,024 audio samples of context [4]. Causality (only left-hand taps) is orthogonal and implemented via asymmetric padding.

The DeepLab pattern: atrous spatial pyramid pooling

Apply several parallel 3×3 dilated convs with dilations $D \in \{6, 12, 18, 24\}$ to the same feature map, then fuse. Each branch captures a different scale; together they cover a wide range of context without needing an image pyramid [5].

Summary

Commit these to memory

1. Master formula: $O = \lfloor (I + 2P - D(K-1) - 1)/S \rfloor + 1$
2. Same padding recipe: $P = D(K-1)/2$ for odd $K$ and stride 1 preserves spatial size.
3. Receptive field recursion: $r_\ell = r_{\ell-1} + (k_\ell - 1) d_\ell \prod_{i<\ell} s_i$
4. Stride subsamples, padding preserves size, dilation enlarges the view for free.

Exercises

Conceptual

1. Compute by hand the output spatial size of nn.Conv2d(64, 128, kernel_size=5, padding=2, stride=1, dilation=1) applied to a 56×56 feature map.
2. Repeat with dilation=2, padding=4. What stays the same; what changes?
3. Two stacked 3×3 convolutions (stride 1) have the same receptive field as what singlek×k kernel? How many fewer parameters do the two 3×3s use, per input-output channel pair?
4. Why does padding='same' raise an error when stride>1 in PyTorch? (Hint: the master formula is no longer invertible to a single integer $P$.)

Coding

1. Extend conv2d_dilated to accept stride and padding in addition to dilation. Verify it matches F.conv2d byte-for-byte on random inputs.
2. Implement a function receptive_field(layers) where layers is a list of (k, s, d) tuples and the function returns the final RF using the recursion above. Test it against the playground's 3-conv VGG example (should give 7).
3. Implement padding modes reflect and replicate in pure Python (no np.pad) and compare to the NumPy output.

Challenge

Reproduce the WaveNet dilated-causal-conv stack (dilations $1, 2, 4, \ldots, 512$, kernel size 2, 10 layers) in PyTorch and print the receptive field after each layer. Verify that it doubles each time. The “causal” part means you must pad on the left only — search the PyTorch docs for F.pad if you get stuck.

References

1. Dumoulin, V., & Visin, F. (2016). A guide to convolution arithmetic for deep learning. arXiv:1603.07285. (Canonical derivation of the output-shape formula.)
2. PyTorch documentation, torch.nn.Conv2d. https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html (Explicit output-size formula and padding_mode options.)
3. Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2015). Striving for Simplicity: The All-Convolutional Net. ICLR workshop. arXiv:1412.6806.
4. van den Oord, A. et al. (2016). WaveNet: A Generative Model for Raw Audio. arXiv:1609.03499.
5. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. (2018). DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE TPAMI 40(4). arXiv:1606.00915.
6. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR. arXiv:1512.03385.
7. Islam, M. A., Jia, S., & Bruce, N. D. B. (2020). How Much Position Information Do Convolutional Neural Networks Encode? ICLR. arXiv:2001.08248.
8. Yu, F., & Koltun, V. (2016). Multi-Scale Context Aggregation by Dilated Convolutions. ICLR. arXiv:1511.07122.
9. Araujo, A., Norris, W., & Sim, J. (2019). Computing Receptive Fields of Convolutional Neural Networks. Distill. https://distill.pub/2019/computing-receptive-fields/
10. Simonyan, K., & Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition (VGG). ICLR. arXiv:1409.1556.

In the next section we'll combine every knob from this chapter into a single production-grade conv2d implementation — including the im2col trick that turns a quadruple loop into a single matrix multiplication.