Convolution

Learning Objectives

By the end of this section, you will be able to:

1. Define the convolution integral $(f * g)(t) = \int_0^t f(\tau) g(t-\tau) \, d\tau$ and explain what each piece represents
2. State and apply the Convolution Theorem: $\mathcal{L}\{f * g\} = F(s) \cdot G(s)$
3. Use convolution to solve initial value problems that are difficult to handle with partial fractions
4. Compute system response of linear time-invariant (LTI) systems via convolution with the impulse response
5. Connect convolution to modern applications in signal processing, convolutional neural networks, and scientific computing

The Big Picture: Why Convolution Exists

"Convolution answers the fundamental question: if a system remembers everything it has ever experienced, what is the total accumulated effect right now?"

Imagine pouring dye into a flowing river. At time $\tau = 0$ you add a drop. At $\tau = 1$, another. At $\tau = 2$, a larger amount. The current concentration of dye at time $t$ depends on every past addition, each decayed by how long ago it happened. That accumulated total is precisely what the convolution integral computes.

In mathematical terms, convolution captures the weighted accumulation of one function over the history of another. It answers:

• Engineering: If I know how a circuit responds to an impulse, what happens when I apply an arbitrary input?
• Physics: Given a time-varying force on a spring, what is the resulting motion?
• Probability: What is the distribution of the sum of two independent random variables?
• Signal processing: How does a filter modify a signal passing through it?
• Machine learning: How does a convolutional layer extract features from data?

The Convolution Theorem provides the crucial bridge: this complex integral in the time domain becomes simple multiplication in the Laplace (or frequency) domain. This duality is one of the most powerful ideas in all of applied mathematics.

Historical Context: Three Centuries of Convolution

The idea of convolution has roots stretching back to the 18th century, though the term "convolution" was not coined until much later.

In the 1750s, Leonhard Euler and Joseph-Louis Lagrange encountered convolution-like integrals while studying the superposition principle for differential equations. When a system is linear, the response to a sum of inputs is the sum of individual responses—an idea that naturally leads to integrating over all past influences.

Pierre-Simon Laplace formalized these ideas around 1782 when he developed his transform method. He showed that certain integrals over products of functions—what we now call convolutions—become simple products in the transformed domain. This was the first incarnation of the Convolution Theorem.

The modern notation and theory were refined by Vito Volterrain the early 1900s (who studied integral equations), and the concept was systematized by Gustav Doetsch who established much of modern Laplace transform theory in the 1930s.

The Convolution Integral

The convolution of two functions $f(t)$ and $g(t)$, both defined for $t \geq 0$, is denoted $(f * g)(t)$ and defined as:

Reading the Integral Symbol by Symbol

Let us decode every piece of this definition:

Notice the key structure: as $\tau$ increases from 0 to $t$, the argument of $f$ moves forward in time while the argument of $g$ moves backward. When $\tau = 0$, we evaluate $f(0) \cdot g(t)$; when $\tau = t$, we evaluate $f(t) \cdot g(0)$. The integral sweeps through all combinations where the arguments sum to $t$.

Properties of Convolution

Convolution obeys several important algebraic properties that mirror those of multiplication. These properties are essential for both theoretical analysis and practical computation.

Why Commutativity Matters

The fact that $f * g = g * f$ means we can compute the convolution integral in whichever order is easier. This is proven by the substitution $u = t - \tau$:

The Delta Function as Identity

The property $f * \delta = f$ is perhaps the most important. By the sifting property of the delta function:

This is why the delta function is called the "identity for convolution"—convolving any function with $\delta(t)$ returns the function unchanged. In system theory, this means: the response to an impulse fully characterizes the system.

Numerical Walkthrough You Can Do By Hand

Before turning to the visualizer, let us evaluate one full convolution manually. Pick $f(t) = e^{-2t}$ and $g(t) = e^{-3t}$ and ask: $(f * g)(0.5) = \;?$

We will compute the same number three different ways — direct integration, the Convolution Theorem, and a four-rectangle Riemann sum — and confirm they agree. Try it on paper first, then open the panel to check.

Interactive: Convolution in Action

Watch the convolution integral being computed in real time. Select two functions and slide the time parameter to see how the product $f(\tau) \cdot g(t - \tau)$ changes and how its integral (the shaded area) traces out the convolution:

You can also explore how different distribution shapes convolve in this broader visualization that uses the "flip and slide" interpretation:

The Convolution Theorem

The Convolution Theorem is the central result that connects convolution in the time domain to multiplication in the Laplace domain. It is arguably the most practically useful theorem in all of Laplace transform theory.

This theorem says: to find the Laplace transform of a convolution, just multiply the individual transforms. Conversely, when you encounter a product $F(s) \cdot G(s)$ in the s-domain and partial fractions are inconvenient, you can find its inverse by computing the convolution of $f(t)$ and $g(t)$.

Proof of the Convolution Theorem

The proof is an elegant application of switching the order of integration. We need to show that $\mathcal{L}\{(f * g)(t)\} = F(s) \cdot G(s)$.

Interactive: The Convolution Theorem

Explore specific examples showing how the Convolution Theorem converts s-domain products into time-domain convolutions:

Solving Initial Value Problems Using Convolution

The Convolution Theorem provides an alternative method for finding inverse Laplace transforms—one that is especially useful when the s-domain expression is a product $F(s) \cdot G(s)$ where partial fractions would be tedious.

The General Strategy

1. Take the Laplace transform of the ODE to get an algebraic equation for $Y(s)$
2. Solve for $Y(s)$ and identify it as a product $F(s) \cdot G(s)$
3. Recognize $f(t) = \mathcal{L}^{-1}\{F(s)\}$ and $g(t) = \mathcal{L}^{-1}\{G(s)\}$
4. Compute the convolution $y(t) = (f * g)(t) = \int_0^t f(\tau)\,g(t-\tau)\,d\tau$

Example: Second-Order IVP

Worked Examples

Example 1: Convolving Two Exponentials

Example 2: Step Function Convolved with Exponential

Example 3: Two Unit Steps

Example 4: Expressing Solutions as Convolutions

System Response and LTI Systems

One of the most important applications of convolution is in the theory of linear time-invariant (LTI) systems. This framework applies to electrical circuits, mechanical systems, control systems, and even neural networks.

The Key Idea

An LTI system is fully characterized by its impulse response $h(t)$—the output when the input is a unit impulse $\delta(t)$. Once you know $h(t)$, the output $y(t)$ for any input $x(t)$ is:

In the Laplace domain, this becomes the beautifully simple relationship:

Physical Examples

Interactive: LTI System Explorer

Explore how different systems respond to different inputs. Select a system type and input signal to see the convolution in action. The output $y(t) = x(t) * h(t)$ is computed numerically:

Machine Learning Connections

Convolution is not just a mathematical curiosity—it is the foundational operation of some of the most successful machine learning architectures ever built. Understanding the calculus of convolution illuminates why these methods work.

Convolutional Neural Networks (CNNs)

In a CNN, each layer applies a set of learned convolution filters to extract features from input data. The operation is:

This is discrete convolution (technically cross-correlation, but the filter is learned so the distinction is moot). The key insight from calculus:

• Feature extraction = convolution: Edge detectors, texture recognizers, and pattern matchers are all convolution filters
• Backpropagation through conv layers involves computing the convolution of the error gradient with the transposed filter
• The Convolution Theorem enables FFT acceleration: For large filters, computing convolution via frequency-domain multiplication is faster than direct computation

The Convolution Theorem and Fast Training

The Convolution Theorem states that convolution can be computed as:

1. Transform both signals to the frequency domain (FFT): $O(n \log n)$
2. Multiply pointwise: $O(n)$
3. Transform back (inverse FFT): $O(n \log n)$

Total: $O(n \log n)$ instead of the $O(n^2)$ of direct convolution. For large signals and filters, this can mean orders of magnitude speedup.

Signal Processing in Audio ML

Audio ML models (speech recognition, music generation) process signals that are continuous-time phenomena sampled at discrete intervals. Understanding the continuous convolution integral helps design:

• Spectral analysis: Understanding frequency content via Fourier/Laplace transforms
• Filter design: Creating low-pass, high-pass, and band-pass filters as convolution kernels
• Reverb modeling: Room acoustics are modeled as convolution with the room's impulse response

Gaussian Processes and Kernel Methods

In Gaussian processes and kernel methods, the convolution of two kernel functions defines a new kernel. The Convolution Theorem provides the spectral characterization: the power spectrum of the convolved kernel is the product of individual power spectra.

Python Implementation

Convolution and the Convolution Theorem

Let's implement convolution both symbolically and numerically, and verify the Convolution Theorem:

1import numpy as np
2from scipy import signal
3
4# Two causal exponentials defined for t >= 0
5def f(t):
6    return np.exp(-2.0 * t)
7
8def g(t):
9    return np.exp(-3.0 * t)
10
11# A fine time grid; dt is the integration step
12dt = 0.001
13t = np.arange(0.0, 4.0, dt)
14
15# Sample the two functions on the grid
16fv = f(t)
17gv = g(t)
18
19# Riemann-sum approximation of the convolution integral
20# (f * g)(t_k) ~ sum_{i=0..k} f(t_i) * g(t_k - t_i) * dt
21conv_num = signal.convolve(fv, gv, mode="full")[: len(t)] * dt
22
23# Closed-form answer from the Convolution Theorem:
24#   F(s) = 1/(s+2),  G(s) = 1/(s+3)
25#   F(s)*G(s) = 1/((s+2)(s+3)) = 1/(s+2) - 1/(s+3)
26#   L^-1{F*G} = e^(-2t) - e^(-3t)
27conv_exact = np.exp(-2.0 * t) - np.exp(-3.0 * t)
28
29# Compare numerical vs exact at t = 0.5 and t = 1.0
30for t_query in (0.5, 1.0):
31    k = int(round(t_query / dt))
32    print(f"t={t_query:.2f}  numerical={conv_num[k]:.6f}"
33          f"  exact={conv_exact[k]:.6f}"
34          f"  error={abs(conv_num[k] - conv_exact[k]):.2e}")

Convolution in Machine Learning

See how convolution appears in signal processing and neural networks:

1import numpy as np
2import matplotlib.pyplot as plt
3
4def convolution_in_ml():
5    """
6    Convolution is foundational to modern ML:
7    - CNNs use discrete 2D convolution for feature extraction
8    - Signal processing in audio ML uses 1D convolution
9    - The convolution theorem enables FFT-based fast training
10    """
11
12    # 1. 1D Convolution for Signal Processing
13    print("=== 1D Convolution: Signal Smoothing ===")
14
15    # Create a noisy signal
16    np.random.seed(42)
17    t = np.linspace(0, 4*np.pi, 500)
18    clean_signal = np.sin(t) + 0.5*np.sin(3*t)
19    noisy_signal = clean_signal + 0.5*np.random.randn(len(t))
20
21    # Gaussian smoothing kernel
22    kernel_size = 31
23    sigma = 3
24    kernel = np.exp(-np.arange(-(kernel_size//2), kernel_size//2 + 1)**2 / (2*sigma**2))
25    kernel = kernel / kernel.sum()  # Normalize
26
27    # Apply convolution
28    smoothed = np.convolve(noisy_signal, kernel, mode='same')
29
30    fig, axes = plt.subplots(2, 2, figsize=(14, 10))
31
32    axes[0, 0].plot(t, noisy_signal, alpha=0.5, label='Noisy Signal')
33    axes[0, 0].plot(t, smoothed, 'r-', linewidth=2, label='Smoothed (convolved)')
34    axes[0, 0].plot(t, clean_signal, 'g--', linewidth=1, label='True Signal')
35    axes[0, 0].set_title('1D Convolution: Signal Denoising')
36    axes[0, 0].legend()
37    axes[0, 0].grid(True, alpha=0.3)
38
39    # 2. Edge Detection (1D derivative kernel)
40    print("\n=== Edge Detection via Convolution ===")
41
42    # Step signal with edges
43    edge_signal = np.zeros(200)
44    edge_signal[50:100] = 1
45    edge_signal[120:180] = 0.5
46
47    # Derivative kernel (edge detector)
48    derivative_kernel = np.array([-1, 0, 1])  # Central difference
49
50    edges = np.convolve(edge_signal, derivative_kernel, mode='same')
51
52    axes[0, 1].plot(edge_signal, 'b-', linewidth=2, label='Signal')
53    axes[0, 1].plot(edges, 'r-', linewidth=2, label='Edges (derivative)')
54    axes[0, 1].set_title('Edge Detection = Convolution with Derivative')
55    axes[0, 1].legend()
56    axes[0, 1].grid(True, alpha=0.3)
57
58    # 3. FFT-based Fast Convolution
59    print("\n=== FFT-based Convolution (Convolution Theorem) ===")
60    print("The Convolution Theorem enables O(n log n) convolution!")
61
62    n = len(noisy_signal)
63
64    # Direct convolution: O(n²)
65    direct_result = np.convolve(noisy_signal, kernel, mode='same')
66
67    # FFT-based convolution: O(n log n)
68    # Pad kernel to same length as signal
69    padded_kernel = np.zeros(n)
70    padded_kernel[:kernel_size] = kernel
71    padded_kernel = np.roll(padded_kernel, -(kernel_size//2))
72
73    fft_signal = np.fft.fft(noisy_signal)
74    fft_kernel = np.fft.fft(padded_kernel)
75    fft_result = np.real(np.fft.ifft(fft_signal * fft_kernel))
76
77    axes[1, 0].plot(direct_result[:200], 'b-', linewidth=2, label='Direct Conv')
78    axes[1, 0].plot(fft_result[:200], 'r--', linewidth=2, label='FFT Conv')
79    axes[1, 0].set_title('Direct vs FFT Convolution (identical)')
80    axes[1, 0].legend()
81    axes[1, 0].grid(True, alpha=0.3)
82
83    # 4. CNN Feature Map (1D analog)
84    print("\n=== CNN: Learned Convolution Filters ===")
85    print("In a CNN, the 'kernel' is LEARNED during training!")
86
87    # Simulate different learned filters
88    filter_low = np.array([0.1, 0.2, 0.4, 0.2, 0.1])  # Low-pass
89    filter_high = np.array([-0.2, -0.1, 0.6, -0.1, -0.2])  # High-pass
90    filter_pattern = np.array([0.5, -0.5, 0.5, -0.5, 0.5])  # Pattern detector
91
92    feat1 = np.convolve(clean_signal, filter_low, mode='same')
93    feat2 = np.convolve(clean_signal, filter_high, mode='same')
94    feat3 = np.convolve(clean_signal, filter_pattern, mode='same')
95
96    axes[1, 1].plot(t[:200], feat1[:200], label='Low-pass filter')
97    axes[1, 1].plot(t[:200], feat2[:200], label='High-pass filter')
98    axes[1, 1].plot(t[:200], feat3[:200], label='Pattern detector')
99    axes[1, 1].set_title('CNN Feature Maps (Convolution Outputs)')
100    axes[1, 1].legend()
101    axes[1, 1].grid(True, alpha=0.3)
102
103    plt.tight_layout()
104    plt.show()
105
106    # Key insight
107    print("\n=== Key ML-Calculus Connection ===")
108    print("1. CNNs learn convolution filters via gradient descent")
109    print("2. The Convolution Theorem enables FFT-based fast training")
110    print("3. Backprop through conv layers uses cross-correlation")
111    print("4. Laplace/Fourier analysis explains filter behavior")
112
113convolution_in_ml()

From the Laplace Integral to a CNN Layer (PyTorch)

The Laplace convolution $(x * h)(t) = \int_0^t x(\tau) h(t-\tau)\,d\tau$ and the discrete convolution $y[k] = \sum_{i} x[k+i]\,h[K-1-i]$ are the same operation in two different worlds — continuous time vs. evenly sampled time. The next snippet runs the discrete version with PyTorch's F.conv1d, then re-derives the answer with a hand-written loop. Both routes produce identical numbers, which is how you know the giant CNN inside ResNet or GPT really is computing the convolution integral your differential-equations textbook introduced two centuries ago.

1import torch
2import torch.nn.functional as F
3
4# A discrete input signal (a tiny time-series)
5x = torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0])
6
7# A 3-tap impulse response — the discrete analog of h(t) in y = x * h
8h = torch.tensor([1.0, 2.0, 3.0])
9
10# conv1d expects shape (batch, channels, length)
11x_b = x.view(1, 1, 5)
12h_b = h.view(1, 1, 3)
13
14# PyTorch's conv1d performs cross-correlation, NOT true convolution.
15# For the true convolution sum y[k] = sum_i x[k+i] * h[K-1-i]
16# we flip the kernel along its time axis first.
17h_flipped = torch.flip(h_b, dims=[-1])
18
19# Apply the discrete convolution
20# Output length = 5 - 3 + 1 = 3 (no padding)
21y = F.conv1d(x_b, h_flipped)
22print("conv1d output:", y.squeeze().tolist())
23
24# Confirm against the textbook discrete-convolution sum
25manual = []
26for k in range(3):
27    s = 0.0
28    for i in range(3):
29        s += x[k + i].item() * h[2 - i].item()
30    manual.append(s)
31print("manual sum   :", manual)

Common Mistakes to Avoid

Test Your Understanding

Summary

Convolution is one of the most consequential operations in all of applied mathematics. It connects time-domain behavior to frequency-domain analysis, characterizes linear systems, and underpins modern signal processing and machine learning.

Key Formulas

Key Takeaways

1. Convolution computes accumulated effect: The integral $\int_0^t f(\tau)g(t-\tau)\,d\tau$ sums all past contributions of $f$ weighted by how they decay via $g$.
2. Time convolution = frequency multiplication: The Convolution Theorem converts the integral into the product $F(s) \cdot G(s)$, enabling algebraic computation.
3. Impulse response characterizes systems: For an LTI system, knowing $h(t)$ determines the response to any input via convolution.
4. Convolution is commutative: $f * g = g * f$. Compute in whichever order is simpler.
5. FFT accelerates convolution: The Convolution Theorem enables $O(n \log n)$ computation instead of $O(n^2)$.
6. CNNs are built on convolution: Feature extraction in deep learning is discrete convolution with learned filters.

Coming Next: In Transfer Functions, we'll see how the ratio $H(s) = Y(s)/X(s)$ completely characterizes the input-output behavior of a linear system, connecting convolution to the powerful framework of block diagrams and feedback control.