Transposed Convolutions

Introduction

Every operator we have met so far in this chapter reduces spatial resolution: a 3×3 conv with stride 1 shrinks the feature map by 2 pixels, a 2×2 max-pool halves it. This is fine for classification, where the pipeline is image → features → class. But some of the most interesting networks in deep learning need to go the other way. A GAN generator takes a 100-dim random vector and produces a 64×64 image. A U-Net decoder takes a 7×7 feature map and recovers a 224×224 segmentation mask. A super-resolution model turns a low-resolution photo into a high-resolution one. For all of these, we need a learned upsampler.

The Core Insight: Transposed convolution is the adjoint of a regular convolution — literally the transpose of its im2col matrix. It is NOT the inverse. “Deconvolution,” the name in older papers, is a mathematical misnomer: the operation undoes the shapechange of a convolution but never recovers the original pixel values. Everything interesting about transposed conv follows from this distinction.

In this section we derive transposed convolution from two complementary viewpoints — a sparse-matrix transpose and a scatter-add operation — and show the two views compute identical numbers. We will also meet the famous checkerboard artifact (Odena, Dumoulin & Olah 2016 [Ref 2]) and the two modern fixes that have mostly replaced transposed conv in state-of-the-art generators and super-resolution networks.

Learning Objectives

After working through this section, you will be able to:

1. Derive the transposed-conv output-size formula $O = (I - 1)S - 2P + D(K-1) + \text{output\_padding} + 1$ from the matrix view.
2. Implement transposed conv from scratch in pure NumPy (scatter-add) and verify the result against nn.ConvTranspose2d byte-for-byte.
3. Explain the checkerboard artifact — reproduce the 1-2-4 overlap pattern and predict when it will appear from K, S alone.
4. Use the two modern alternatives: upsample + conv (Odena) and pixel shuffle (Shi et al. [4]).
5. Read and write DCGAN, U-Net and FCN decoder blocks with confidence.
6. Know when to use output_padding and why PyTorch requires it to be less than the stride.

Why We Need Learned Upsampling

Three classes of vision architecture are fundamentally dilative: they start small and end large, or they need to reconstruct fine detail from coarse representations.

You could use torch.nn.Upsample(mode='bilinear') for every one of these and add a stride-1 conv afterwards. That is in fact a perfectly reasonable modern design (and we will see why below). Historically, though, each new upsampling step was implemented as one transposed-conv layer, so the weights had to learn both the resize and the filter at once. We need to understand transposed conv to read a decade of papers, state dicts, and reference implementations.

A Name Problem — Deconvolution, Transposed, Fractionally-Strided

The operation has three names in the literature. They all refer to the same thing but carry different baggage.

The Matrix View — It Really Is a Transpose

From §10.4 we know that a forward convolution can be written as $\mathbf{y} = W\mathbf{x}$, where $W$ is the sparse im2col matrix and $\mathbf{x}$ is the flattened input. For a 3×3 kernel mapping a 4×4 image to a 2×2 output, $W$ is a $(4, 16)$ matrix with 9 non-zeros per row.

The transposed convolution with the same kernel is, by definition:

$\mathbf{y}' = W^{\top} \mathbf{x}'$

where $\mathbf{x}'$ is now the flattened 2×2 input (length 4) and $\mathbf{y}'$ is the 4×4 output (length 16). $W^{\top}$ has shape $(16, 4)$. Every row corresponds to one output cell and is populated by a specific subset of the 9 kernel taps.

Interactive: Transpose as Adjoint

Hover any cell of the 2×2 input or the 4×4 output below. The column of $W^{\top}$ belonging to that input cell lights up; so do the output cells it paints into. Hovering an output cell highlights its row of $W^{\top}$ and the input cells whose kernel taps feed it. The adjoint relationship is no longer a definition — it's a click.

The Scatter-Add View — Hand Computation

Multiplying a 16×4 matrix by a 4-vector is the mathematical definition, but nobody actually implements transposed conv that way. The practical trick: every input cell $x[i,j]$ “paints” a scaled copy of the kernel into the output, starting at position $(i \cdot S, j \cdot S)$, and overlapping copies accumulate. This is exactly the rule you get by reading $W^{\top} \mathbf{x}$ column by column.

Worked example — 2×2 input, 3×3 kernel, stride 1

Let us paint by hand. Our input is the 2×2 tensor [[1, 2], [3, 4]] and our kernel is the sparse [[1, 0, 1], [0, 1, 0], [1, 0, 1]]. Stride 1, padding 0. The output size is $(2-1)\cdot 1 + 3 = 4$, so the result is 4×4.

Summing overlapping contributions gives:

1[[1, 2, 1, 2],
2 [3, 5, 5, 4],
3 [1, 5, 5, 2],
4 [3, 4, 3, 4]]

Try e.g. cell (1, 1) — it is hit by four of the five non-zero kernel taps from four different input cells: $1 \cdot 1 + 2 \cdot 0 + 3 \cdot 0 + 4 \cdot 1 = 5$. ✓

Interactive: Scatter-Add Painter

Click an input cell or press Play to watch every input cell paint a scaled copy of the kernel into the output. Switch the kernel between “sparse corners-plus-centre” (the example above) and all-ones, and try strides 1, 2, 3 to see how zero-stride windows cease to overlap. The numbers in green are running sums of the contributions you have painted so far — once all four input cells are painted, you should see the same [[1,2,1,2],[3,5,5,4],[1,5,5,2],[3,4,3,4]] the table predicts.

Transposed Conv in Pure Python

    c0 c1
r0   1  2
r1   3  4

    c0 c1 c2
r0   1  0  1
r1   0  1  0
r2   1  0  1

[[1. 2. 1. 2.]
 [3. 5. 5. 4.]
 [1. 5. 5. 2.]
 [3. 4. 3. 4.]]

1import numpy as np
2
3# Input feature map (2x2)
4x = np.array([
5    [1, 2],
6    [3, 4],
7], dtype=float)
8
9# Kernel (3x3) — sparse 'corners+center' so we can verify by hand
10w = np.array([
11    [1, 0, 1],
12    [0, 1, 0],
13    [1, 0, 1],
14], dtype=float)
15
16def conv_transpose2d(x, w, stride=1, padding=0):
17    """Transposed 2-D convolution via the scatter-add interpretation.
18
19    For every input cell x[i,j], add x[i,j] * w to the output region
20    starting at (i*stride, j*stride). Then strip 'padding' pixels off
21    each side of the result.
22    """
23    H_in, W_in = x.shape
24    kH, kW = w.shape
25    H_big = (H_in - 1) * stride + kH
26    W_big = (W_in - 1) * stride + kW
27    y_big = np.zeros((H_big, W_big))
28
29    for i in range(H_in):
30        for j in range(W_in):
31            r = i * stride
32            c = j * stride
33            y_big[r:r+kH, c:c+kW] += x[i, j] * w
34
35    # The padding P in the forward conv becomes a CROP P in the transpose
36    if padding > 0:
37        y_big = y_big[padding:-padding, padding:-padding]
38    return y_big
39
40print(conv_transpose2d(x, w, stride=1, padding=0))
41# [[1. 2. 1. 2.]
42#  [3. 5. 5. 4.]
43#  [1. 5. 5. 2.]
44#  [3. 4. 3. 4.]]

Equivalence with the Transposed Matrix

The two views must agree numerically. Let us build $W$ explicitly and verify:

1import numpy as np
2
3# Same 2x2 input and 3x3 kernel as before
4x = np.array([[1., 2.], [3., 4.]])
5w = np.array([[1., 0., 1.],
6              [0., 1., 0.],
7              [1., 0., 1.]])
8
9# STEP 1 — build the (4, 16) im2col-style matrix W for the FORWARD
10# conv mapping a 4x4 image to a 2x2 output. Each row of W picks out
11# the 3x3 window that produces one output cell.
12def forward_conv_matrix(kH, kW, H_in, W_in):
13    H_out = H_in - kH + 1
14    W_out = W_in - kW + 1
15    rows = H_out * W_out
16    cols = H_in * W_in
17    W_mat = np.zeros((rows, cols))
18    for i in range(H_out):
19        for j in range(W_out):
20            r = i * W_out + j
21            for di in range(kH):
22                for dj in range(kW):
23                    c = (i + di) * W_in + (j + dj)
24                    W_mat[r, c] = w[di, dj]
25    return W_mat
26
27W_fwd = forward_conv_matrix(3, 3, 4, 4)   # shape (4, 16)
28print("forward matrix shape:", W_fwd.shape)
29
30# STEP 2 — transposed conv IS multiplication by W_fwd.T
31x_flat = x.flatten()                       # shape (4,)
32y_flat = W_fwd.T @ x_flat                  # shape (16,)
33y_matrix_view = y_flat.reshape(4, 4)
34
35print("matrix-transpose view:")
36print(y_matrix_view)
37# [[1. 2. 1. 2.]
38#  [3. 5. 5. 4.]
39#  [1. 5. 5. 2.]
40#  [3. 4. 3. 4.]]
41
42# Must match the scatter-add result byte-for-byte:
43from numpy.testing import assert_allclose
44# (assuming conv_transpose2d from the previous block is in scope)
45# assert_allclose(conv_transpose2d(x, w), y_matrix_view)

Transposed Conv in PyTorch

1import torch
2import torch.nn as nn
3import torch.nn.functional as F
4
5# Same 2x2 input, same 3x3 kernel, but now in (N, C_in, H, W) layout
6x = torch.tensor([[[
7    [1., 2.],
8    [3., 4.],
9]]])  # shape (1, 1, 2, 2)
10
11# Build the ConvTranspose2d module and manually set its weights so we
12# can cross-check against the NumPy result.
13ct = nn.ConvTranspose2d(
14    in_channels=1,
15    out_channels=1,
16    kernel_size=3,
17    stride=1,
18    padding=0,
19    bias=False,
20)
21with torch.no_grad():
22    ct.weight.copy_(torch.tensor([[[
23        [1., 0., 1.],
24        [0., 1., 0.],
25        [1., 0., 1.],
26    ]]]))
27
28y = ct(x)
29print(y.squeeze())
30# tensor([[1., 2., 1., 2.],
31#         [3., 5., 5., 4.],
32#         [1., 5., 5., 2.],
33#         [3., 4., 3., 4.]])
34
35# Functional form
36y2 = F.conv_transpose2d(
37    x,
38    ct.weight,
39    stride=1,
40    padding=0,
41)
42assert torch.equal(y, y2)

The Output-Size Formula

The full formula, from Dumoulin & Visin (2016) [1], Eq. 15, and repeated verbatim in the PyTorch docs for torch.nn.ConvTranspose2d:

$O = (I - 1) S - 2P + D(K - 1) + \text{output\_padding} + 1$

With each symbol:

1import torch
2import torch.nn.functional as F
3
4def convT_out(I, K, S, P, output_padding=0, dilation=1):
5    """Dumoulin & Visin (2016), Eq. 15, reproduced verbatim from the PyTorch
6    torch.nn.ConvTranspose2d docs."""
7    return (I - 1) * S - 2 * P + dilation * (K - 1) + output_padding + 1
8
9# Verify against PyTorch on 20 random configurations
10torch.manual_seed(0)
11for _ in range(20):
12    I = torch.randint(2, 20, (1,)).item()
13    K = torch.randint(1, 6, (1,)).item()
14    S = torch.randint(1, 4, (1,)).item()
15    P = torch.randint(0, K, (1,)).item()
16    op = torch.randint(0, S, (1,)).item()  # output_padding < stride (PyTorch requires this)
17
18    kernel = torch.randn(1, 1, K, K)
19    x = torch.randn(1, 1, I, I)
20    y = F.conv_transpose2d(x, kernel, stride=S, padding=P, output_padding=op)
21
22    predicted = convT_out(I, K, S, P, op)
23    actual = y.shape[-1]
24    assert predicted == actual, (I, K, S, P, op, predicted, actual)
25print("All 20 random configurations match the formula.")

Interactive: Output-Size Calculator

Drag the six sliders to see every term of the output formula light up as it is added. The forward-conv counterpart range tells you which forward-conv inputs all collapse to the current $I$, and output_padding is automatically clamped to $< S$ (PyTorch's constraint). The copy-paste-ready PyTorch invocation is the same line you would type when reproducing this configuration in code.

Strided Transposed Convolution — Zero Insertion

When $S > 1$, transposed conv becomes an upsampler. The classic “fractionally-strided” interpretation makes this intuitive: insert $S-1$ zero rows and columns between every pair of input cells, pad the result by $K - 1 - P$ zeros on each side, and then apply a regular stride-1 convolution with the same kernel. The output size matches the formula above.

Example — 2×2 input, K=3, S=2:

1Input (2x2):
2    [[1, 2],
3     [3, 4]]
4
5After inserting S-1 = 1 zero between cells (3x3):
6    [[1, 0, 2],
7     [0, 0, 0],
8     [3, 0, 4]]
9
10Pad by K-1-P = 2 zeros around (7x7):
11    [[0, 0, 0, 0, 0, 0, 0],
12     [0, 0, 0, 0, 0, 0, 0],
13     [0, 0, 1, 0, 2, 0, 0],
14     [0, 0, 0, 0, 0, 0, 0],
15     [0, 0, 3, 0, 4, 0, 0],
16     [0, 0, 0, 0, 0, 0, 0],
17     [0, 0, 0, 0, 0, 0, 0]]
18
19Apply regular 3x3 conv, stride 1 → 5x5 output.
20(Output size check: (2-1)*2 + 3 = 5 ✓)

Modern implementations skip the explicit zero insertion (too wasteful) and go straight to the scatter-add form or to a cuDNN routine that fuses both steps. But the zero-insertion picture is useful for intuition and is the origin of the name “fractionally-strided convolution”.

Interactive: Fractionally-Strided View

Step through the four stages below — original input, zero-inserted, zero-padded, then a regular stride-1 conv slid across the result. The output size of the final conv equals the transposed-conv output size from the formula above, exactly. Toggle stride and padding to confirm.

output_padding — Resolving the Ambiguity

A forward conv with $S > 1$ does floor-division in its output formula. Two adjacent input sizes can map to the same output size, so the transposed conv going the other way is ambiguous: given a 4×4 output, was the forward input 7×7 or 8×8? output_padding picks the answer.

1import torch
2import torch.nn as nn
3import torch.nn.functional as F
4
5# Two DIFFERENT forward convs that produce the SAME output size:
6#   input 7,  K=3, S=2, P=1  →  (7 + 2 - 3) / 2 + 1 = 4 (floor)
7#   input 8,  K=3, S=2, P=1  →  (8 + 2 - 3) / 2 + 1 = 4 (integer)
8# Going in reverse, we get a 4x4 input and want either 7x7 OR 8x8 back.
9# The ambiguity is resolved by output_padding.
10
11x = torch.randn(1, 1, 4, 4)
12ct = nn.ConvTranspose2d(1, 1, kernel_size=3, stride=2, padding=1, bias=False)
13
14# output_padding=0 → smaller of the two candidates (7x7)
15y_small = F.conv_transpose2d(x, ct.weight, stride=2, padding=1, output_padding=0)
16print("output_padding=0:", y_small.shape)  # torch.Size([1, 1, 7, 7])
17
18# output_padding=1 → larger candidate (8x8)
19y_large = F.conv_transpose2d(x, ct.weight, stride=2, padding=1, output_padding=1)
20print("output_padding=1:", y_large.shape)  # torch.Size([1, 1, 8, 8])
21
22# Round-trip check: forward conv on the 8x8 output maps back to 4x4
23x_back = F.conv2d(y_large, ct.weight, stride=2, padding=1)
24print("round-trip:", x_back.shape)          # torch.Size([1, 1, 4, 4])
25# NOTE: values differ — transposed conv is NOT the inverse of convolution.
26# Only the SHAPE round-trips; the values would require w and w^+ (pseudo-inverse) to match.

The Checkerboard Artifact — Odena, Dumoulin & Olah (2016)

The most important known pathology of transposed convolution. Odena, Dumoulin & Olah (2016) [Ref 2], in a short and influential Distill article, showed that GAN generators built from stacked ConvTranspose2d(K=3, S=2) layers produce outputs with visible grid-like patterns — even when the training signal contains no such patterns. The cause is a purely geometric property of transposed conv, not a training bug.

Reproducing the 1-2-4 Overlap Pattern

Take a uniform 3×3 input of ones, a uniform 3×3 kernel of ones, and apply stride-2 transposed conv. The output — which should be uniform by symmetry — is not.

[[1 1 2 1 2 1 1]
 [1 1 2 1 2 1 1]
 [2 2 4 2 4 2 2]
 [1 1 2 1 2 1 1]
 [2 2 4 2 4 2 2]
 [1 1 2 1 2 1 1]
 [1 1 2 1 2 1 1]]

1import numpy as np
2
3# Odena, Dumoulin & Olah (2016) — "Deconvolution and Checkerboard Artifacts"
4# Setup: 3x3 feature map of all ones, K=3 kernel of all ones, stride=2.
5x = np.ones((3, 3))
6w = np.ones((3, 3))
7
8def conv_transpose2d_strided(x, w, stride):
9    H_in, W_in = x.shape
10    kH, kW = w.shape
11    H_out = (H_in - 1) * stride + kH
12    W_out = (W_in - 1) * stride + kW
13    y = np.zeros((H_out, W_out))
14    for i in range(H_in):
15        for j in range(W_in):
16            y[i*stride:i*stride+kH, j*stride:j*stride+kW] += x[i, j] * w
17    return y
18
19print(conv_transpose2d_strided(x, w, stride=2))
20# [[1. 1. 2. 1. 2. 1. 1.]
21#  [1. 1. 2. 1. 2. 1. 1.]
22#  [2. 2. 4. 2. 4. 2. 2.]
23#  [1. 1. 2. 1. 2. 1. 1.]
24#  [2. 2. 4. 2. 4. 2. 2.]
25#  [1. 1. 2. 1. 2. 1. 1.]
26#  [1. 1. 2. 1. 2. 1. 1.]]
27
28# The periodic 1-2-4 pattern is the CHECKERBOARD. Even with a perfectly
29# uniform input and a perfectly uniform kernel, transposed-conv output
30# is non-uniform whenever kernel_size is not divisible by stride.

Why does this happen? Each output cell $(r, c)$ is the sum of contributions from every input cell whose scattered kernel touches it. With $K=3$ and $S=2$, the number of input cells that touch a given output cell alternates between 1 and 2 along each axis, producing the 1-2-4 pattern we see. Whenever $K$ is not a multiple of $S$, the overlap is non-uniform and a checkerboard appears.

Interactive: Overlap Heatmap Explorer

Slide $K$ and $S$ below. Whenever $K \bmod S = 0$, the heatmap is a single uniform colour and the diagnostic strip turns green; otherwise it shows the periodic checkerboard pattern. This is exactly Figure 3 from Odena, Dumoulin & Olah (2016) [Ref 2], but you can drag the knobs.

Fix 1 — Nearest Upsample + Regular Conv

Odena's recommended fix: decouple upsampling from filtering. First use a parameter-free nearest-neighbour (or bilinear) nn.Upsample to resize, then apply a regular nn.Conv2d to filter. Same receptive field, same parameter count, no checkerboard.

1import torch
2import torch.nn as nn
3import torch.nn.functional as F
4
5# Same (N=1, C_in=4, H=7, W=7) input for both variants
6x = torch.randn(1, 4, 7, 7)
7
8# Variant A — the classic (prone to checkerboard)
9convT = nn.ConvTranspose2d(4, 8, kernel_size=3, stride=2, padding=1, output_padding=1)
10yA = convT(x)
11print("convT:        ", yA.shape)  # torch.Size([1, 8, 14, 14])
12
13# Variant B — Odena et al. (2016) recommendation:
14#             nearest-upsample, then stride-1 conv
15class UpsampleConv(nn.Module):
16    def __init__(self, C_in, C_out, scale_factor=2, kernel_size=3, padding=1):
17        super().__init__()
18        self.up = nn.Upsample(scale_factor=scale_factor, mode='nearest')
19        self.conv = nn.Conv2d(C_in, C_out, kernel_size=kernel_size, padding=padding)
20    def forward(self, x):
21        return self.conv(self.up(x))
22
23up_conv = UpsampleConv(4, 8)
24yB = up_conv(x)
25print("upsample+conv:", yB.shape)  # torch.Size([1, 8, 14, 14])
26
27# Same shape, roughly same parameter count (9·4·8 + 8 = 296 either way),
28# but the upsample+conv variant eliminates the 1-2-4 overlap pattern.

Fix 2 — Pixel Shuffle / Sub-Pixel Convolution

Shi et al. (2016) [Ref 4]'s ESPCN introduced a different approach: do the convolution in low resolution with $r^2 C_\text{out}$ output channels, then use a zero-parameter pixel-shuffle to permute those channels into $r$-times larger spatial extent.

Interactive: PixelShuffle Rearrangement

Hover any cell on either side of the visualizer below. The mapping rule $y[c, ri+di, rj+dj] = x[c + C(di\,r + dj),\ i,\ j]$ becomes click-tracking; the $r^2$ input channels collapse onto exactly $r^2$ output pixels per super-pixel, with no overlap — which is why pixel shuffle is artifact-free by construction.

1import torch
2import torch.nn as nn
3
4# Pixel shuffle (Shi et al. 2016 [4]) — aka 'sub-pixel convolution'.
5# Key idea: produce r^2 * C_out output channels with a regular conv,
6# then rearrange them into one tensor with r^2 times the spatial area.
7
8class SubPixelUp(nn.Module):
9    """Replace ConvTranspose2d with conv → pixel_shuffle. No checkerboard."""
10    def __init__(self, C_in, C_out, scale_factor=2):
11        super().__init__()
12        r = scale_factor
13        # Conv produces r^2 * C_out channels at the SAME spatial size
14        self.conv = nn.Conv2d(C_in, C_out * r * r, kernel_size=3, padding=1)
15        # PixelShuffle rearranges (r^2 * C_out, H, W) → (C_out, r*H, r*W)
16        self.shuffle = nn.PixelShuffle(upscale_factor=r)
17
18    def forward(self, x):
19        return self.shuffle(self.conv(x))
20
21x = torch.randn(1, 4, 7, 7)
22up = SubPixelUp(C_in=4, C_out=8, scale_factor=2)
23y = up(x)
24print(y.shape)  # torch.Size([1, 8, 14, 14])
25
26# Parameter count — same 3x3 conv as before, just with 4x more output
27# channels (8 * 4 = 32), so 9*4*32 + 32 = 1184 parameters.
28n_params = sum(p.numel() for p in up.parameters())
29print(f"params: {n_params}")  # 1184

The Backward Pass — It Really Is a Regular Convolution

§10.4 derived the backward pass of a forward conv: $\partial L / \partial \mathbf{x}$ is itself a convolution of $\partial L / \partial \mathbf{y}$ with the flipped kernel. The analogous fact for transposed conv is even cleaner: because the forward pass is $\mathbf{y} = W^{\top} \mathbf{x}$, differentiating gives:

$\frac{\partial L}{\partial \mathbf{x}} = W \cdot \frac{\partial L}{\partial \mathbf{y}}$

But $W \cdot \mathbf{v}$ is exactly the forward pass of a regular convolution. So:

The backward pass of ConvTranspose2d is a forward Conv2d. And the backward pass of Conv2dis a forward ConvTranspose2d. The two operators are each other's backward pass — which is why PyTorch implements them with shared cuDNN kernels.

This duality is the reason they have the same parameter count for matching hyperparameters, and the reason cuDNN can reuse the same highly-optimised im2col-and-GEMM primitives (§10.4) to accelerate both.

Applications in the Wild

DCGAN Generator

Radford, Metz & Chintala (2016) [Ref 3] introduced the deep convolutional GAN: a generator built entirely from transposed convolutions stacked to upsample a 100-dim latent vector into a 64×64 RGB image.

1import torch.nn as nn
2
3class DCGANGenerator(nn.Module):
4    """Radford, Metz & Chintala (2016), canonical 64x64 generator."""
5    def __init__(self, nz=100, ngf=64, nc=3):
6        super().__init__()
7        self.main = nn.Sequential(
8            # Latent z: (nz, 1, 1)  →  (ngf*8, 4, 4)
9            nn.ConvTranspose2d(nz, ngf*8, 4, 1, 0, bias=False),
10            nn.BatchNorm2d(ngf*8), nn.ReLU(True),
11            # (ngf*8, 4, 4)  →  (ngf*4, 8, 8)
12            nn.ConvTranspose2d(ngf*8, ngf*4, 4, 2, 1, bias=False),
13            nn.BatchNorm2d(ngf*4), nn.ReLU(True),
14            # (ngf*4, 8, 8)  →  (ngf*2, 16, 16)
15            nn.ConvTranspose2d(ngf*4, ngf*2, 4, 2, 1, bias=False),
16            nn.BatchNorm2d(ngf*2), nn.ReLU(True),
17            # (ngf*2, 16, 16)  →  (ngf, 32, 32)
18            nn.ConvTranspose2d(ngf*2, ngf,   4, 2, 1, bias=False),
19            nn.BatchNorm2d(ngf),   nn.ReLU(True),
20            # (ngf, 32, 32)  →  (nc, 64, 64)
21            nn.ConvTranspose2d(ngf,  nc,     4, 2, 1, bias=False),
22            nn.Tanh(),
23        )
24    def forward(self, z):
25        return self.main(z)

U-Net Decoder

Ronneberger, Fischer & Brox (2015) [Ref 7] introduced U-Net for biomedical segmentation. The encoder halves the spatial resolution stage-by-stage with max-pool (§10.5); the decoder doubles it back with transposed conv, concatenating the matching-resolution encoder feature map at every step via a skip connection.

1import torch, torch.nn as nn
2
3class UNetUpBlock(nn.Module):
4    """One U-Net decoder stage: upsample then fuse with the matching skip."""
5    def __init__(self, C_in, C_out):
6        super().__init__()
7        self.up = nn.ConvTranspose2d(C_in, C_out, kernel_size=2, stride=2)
8        self.conv = nn.Sequential(
9            nn.Conv2d(C_out*2, C_out, 3, padding=1), nn.ReLU(True),
10            nn.Conv2d(C_out,   C_out, 3, padding=1), nn.ReLU(True),
11        )
12
13    def forward(self, x, skip):
14        x = self.up(x)                                   # double spatial
15        x = torch.cat([skip, x], dim=1)                  # concat along channels
16        return self.conv(x)

The K=2, S=2 transposed conv exactly doubles spatial resolution and is checkerboard-safe (2 is a multiple of 2). Modern U-Net variants often replace the transposed conv with nn.Upsample(scale_factor=2) + nn.Conv2d(...)— same shape, no artifacts.

FCN for Semantic Segmentation

Long, Shelhamer & Darrell (2015) [Ref 6] introduced the first fully convolutional network for semantic segmentation. They used transposed conv to upsample the coarse class-score map from the last conv stage back to image resolution, initialising the kernel to bilinear interpolation and then letting the network fine-tune it. This initialisation trick is still used today when a decoder is trained from scratch on small data.

[[0.0625, 0.1875, 0.1875, 0.0625],
 [0.1875, 0.5625, 0.5625, 0.1875],
 [0.1875, 0.5625, 0.5625, 0.1875],
 [0.0625, 0.1875, 0.1875, 0.0625]]

1import torch
2import torch.nn as nn
3
4def bilinear_kernel(C_in, C_out, K):
5    """Bilinear-interpolation kernel (Long et al. 2015, supplementary)."""
6    factor = (K + 1) // 2
7    center = factor - 1 if K % 2 == 1 else factor - 0.5
8    og = torch.arange(K).float()
9    filt1d = 1 - (og - center).abs() / factor
10    filt2d = filt1d.unsqueeze(0) * filt1d.unsqueeze(1)
11    weight = torch.zeros(C_in, C_out, K, K)
12    for c in range(min(C_in, C_out)):
13        weight[c, c] = filt2d
14    return weight
15
16# 32x upsample at the end of FCN-32s
17upsample = nn.ConvTranspose2d(21, 21, kernel_size=64, stride=32,
18                              padding=16, bias=False)
19upsample.weight.data.copy_(bilinear_kernel(21, 21, 64))

Design Patterns — Pick the Right Upsampler

Summary

Commit these to memory

1. Output formula: $O = (I-1)S - 2P + D(K-1) + \text{output\_padding} + 1$
2. Matrix identity: transposed conv $= W^{\top} \mathbf{x}$ where $W$ is the forward im2col matrix. NOT the inverse of convolution.
3. Checkerboard diagnostic: stride-S transposed conv is artifact-free iff kernel size $K$ is a multiple of $S$. Prefer K=2,S=2 or K=4,S=2.
4. The two modern replacements: nearest-upsample + conv (Odena [2]) and conv + pixel-shuffle (Shi [4]). Same shape, same parameter count, no artifacts.
5. Weight-shape gotcha: ConvTranspose2d stores weights as $(C_\text{in}, C_\text{out}, kH, kW)$, opposite to Conv2d.
6. Duality: backward of ConvTranspose2d = forward of Conv2d, and vice versa.

Exercises

Conceptual

1. Compute by hand the output spatial size of nn.ConvTranspose2d(32, 64, kernel_size=4, stride=2, padding=1) applied to a 14×14 feature map.
2. Show that for $K = 2, S = 1, P = 0$, transposed conv and forward conv have the same output size. Why? (Hint: plug into both formulas.)
3. Explain why transposed conv is NOT the inverse of convolution. Construct a small example where $W^{\top} W \mathbf{x} \neq \mathbf{x}$.
4. If forward conv(K=3, S=2, P=1) maps inputs 7 and 8 both to 4, which value of output_padding recovers each?
5. Why do DCGAN generators use K=4, S=2 instead of K=3, S=2? Answer in one sentence referencing Odena et al. (2016) [2].

Coding

1. Extend the scatter-add conv_transpose2d to handle multi-channel inputs and outputs. Verify against F.conv_transpose2d on random 10 configurations.
2. Reproduce Figure 3 from Odena et al. (2016) [2]: plot the overlap-count heatmap for (K, S) pairs in $\{2,3,4\} \times \{1,2,3\}$and verify that uniformity occurs iff K is a multiple of S.
3. Train two tiny image autoencoders on MNIST: one using stacked ConvTranspose2d(K=3, S=2) in the decoder, the other using Upsample + Conv2d. Compare reconstruction quality and plot a few samples side by side. Do you see checkerboards?
4. Implement nn.PixelShuffle(r) from scratch using tensor.reshape and tensor.permute. Verify against the built-in module.

Challenge

Reproduce the Odena et al. (2016) [2] experiment. Train a small DCGAN on CIFAR-10 for 20 epochs twice — once with ConvTranspose2d(K=3, S=2) layers and once with Upsample + Conv2d. Visualise a batch of generated samples from both models and measure the FID score. The original paper shows a ~10–30% FID improvement from eliminating the checkerboard. Reproducing even a qualitative version of this result is the clearest possible demonstration of the theoretical argument.

References

1. Dumoulin, V., & Visin, F. (2016). A guide to convolution arithmetic for deep learning. arXiv:1603.07285. (The canonical derivation of the transposed conv output formula, §4. Figure 4.1 is the 2×2 → 4×4 example we hand-computed.)
2. Odena, A., Dumoulin, V., & Olah, C. (2016). Deconvolution and Checkerboard Artifacts. Distill. https://distill.pub/2016/deconv-checkerboard/ (Diagnoses the checkerboard, proposes upsample-then-conv.)
3. Radford, A., Metz, L., & Chintala, S. (2016). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (DCGAN). ICLR. arXiv:1511.06434.
4. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016). Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network (ESPCN). CVPR. arXiv:1609.05158. (Introduces pixel shuffle.)
5. Zeiler, M. D., Krishnan, D., Taylor, G. W., & Fergus, R. (2010). Deconvolutional Networks. CVPR. (Early use of “deconvolution” for reconstructing inputs from feature maps; source of the misleading name.)
6. Long, J., Shelhamer, E., & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. CVPR. arXiv:1411.4038. (First large-scale use of transposed conv in vision. Introduces bilinear-init trick for the upsample layer.)
7. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI. arXiv:1505.04597. (Encoder pool + decoder transposed-conv + skip connections.)
8. Noh, H., Hong, S., & Han, B. (2015). Learning Deconvolution Network for Semantic Segmentation. ICCV. arXiv:1505.04366. (The “Deconvnet” architecture — representative of the checkerboard-prone K=3, S=2 stack.)
9. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, §9.5 “Variants of the Basic Convolution Function”. MIT Press. https://www.deeplearningbook.org/ (Textbook treatment of transposed conv and its role as the adjoint of ordinary conv.)
10. PyTorch documentation, torch.nn.ConvTranspose2d, torch.nn.functional.conv_transpose2d, torch.nn.Upsample, torch.nn.PixelShuffle. https://pytorch.org/docs/stable/nn.html (Authoritative reference foroutput_padding semantics and the weight-shape ordering.)
11. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020). Analyzing and Improving the Image Quality of StyleGAN (StyleGAN2). CVPR. arXiv:1912.04958. (Moves from transposed-conv to bilinear-upsample-then-conv to eliminate residual artifacts.)

This concludes Chapter 10. You now have a complete, mathematically grounded view of the convolution operator and its entire ecosystem — the forward pass, the parameters, the efficient implementation, pooling, and transposed convolution. Chapter 11 builds on these foundations to walk through the architectural lineage that shaped modern computer vision: LeNet, AlexNet, VGG, Inception, ResNet, DenseNet, EfficientNet.