Boo-AI — Master Artificial Intelligence by Building from Scratch

Learning Objectives

By the end of this section, you will be able to:

Apply the Laplace transform to convert differential equations into algebraic equations
Incorporate initial conditions directly through the derivative property of Laplace transforms
Solve first-order and second-order initial value problems using the Laplace method
Compare the Laplace transform approach with classical time-domain methods
Apply these techniques to real-world problems in circuit analysis and mechanical systems
Recognize how IVP solutions appear in machine learning contexts

The Big Picture: Calculus Without Calculus

"The Laplace transform is a machine that turns the hard problem of solving differential equations into the easier problem of solving algebraic equations."

Differential equations describe how quantities change over time. Finding their solutions traditionally requires integration, finding particular and homogeneous solutions, and determining constants from initial conditions. This can be tedious, especially for higher-order equations.

The Laplace transform offers a remarkable alternative: it converts differentiation into multiplication. Instead of solving a calculus problem, we solve an algebra problem. The key insight is:

The Core Transformation

Time Domain

\\frac{dy}{dt}

Derivative

⟹

s-Domain

sY(s) - y(0)

Multiplication + IC

Notice something remarkable: the initial condition $y(0)$ appears automatically! This is the genius of the Laplace method for initial value problems. We do not find a general solution and then apply ICs; the ICs are built in from the start.

Why This Matters

In applications like circuit analysis, control systems, and mechanical engineering, we always know the initial state (a capacitor's charge, a mass's position). The Laplace transform method naturally incorporates this information, making it the preferred tool for engineers.

Historical Context: From Heaviside to Modern Engineering

The Laplace transform was developed by Pierre-Simon Laplace in the late 18th century for probability theory. However, it was Oliver Heaviside (1850-1925) who recognized its power for solving differential equations in electrical engineering.

Heaviside, a self-taught English mathematician, developed "operational calculus" — treating derivatives as algebraic operations. Though his methods initially lacked rigorous justification, they worked remarkably well. The mathematical community eventually formalized his techniques using the Laplace transform.

The Engineering Perspective

Engineers adopted Laplace transforms enthusiastically because:

Initial conditions are automatic: In real systems, you always know the starting state
System analysis becomes multiplication: Cascading systems multiply their transfer functions
Frequency response is immediate: Setting $s = j\\omega$ gives the frequency response
Tables make it practical: Common transforms are tabulated for quick lookup

The s-Domain

Engineers call the Laplace domain the "s-domain" or "complex frequency domain." The variable $s = \\sigma + j\\omega$ combines exponential decay/growth ( $\\sigma$ ) with oscillation frequency ( $\\omega$ ).

The Laplace Transform of Derivatives

The key property that makes Laplace transforms useful for IVPs is how they handle derivatives. Let's derive the formula for first and second derivatives.

First Derivative

By definition, $\\mathcal{L}\\{y'(t)\\} = \\int_0^\\infty e^{-st} y'(t) \\, dt$ . Using integration by parts with $u = e^{-st}$ and $dv = y'(t)\\,dt$ :

\\mathcal{L}\\{y'(t)\\} = \\left[e^{-st}y(t)\\right]_0^\\infty + s\\int_0^\\infty e^{-st}y(t)\\,dt

The boundary term at $t = \\infty$ vanishes (assuming $y$ doesn't grow too fast), and the term at $t = 0$ gives $-y(0)$ . The integral is just $\\mathcal{L}\\{y(t)\\} = Y(s)$ . Therefore:

First Derivative Property

\\mathcal{L}\\{y'(t)\\} = sY(s) - y(0)

Second Derivative

For the second derivative, we apply the first derivative rule twice:

$\\mathcal{L}\\{y''\\} = s \\cdot \\mathcal{L}\\{y'\\} - y'(0)$

$= s[sY(s) - y(0)] - y'(0)$

Second Derivative Property

\\mathcal{L}\\{y''(t)\\} = s^2Y(s) - sy(0) - y'(0)

The pattern is clear: Each derivative introduces a factor of $s$ , and each initial condition appears with the appropriate power of $s$ .

General n-th Derivative

For completeness, the transform of the $n$ -th derivative is:

\\mathcal{L}\\{y^{(n)}\\} = s^n Y(s) - s^{n-1}y(0) - s^{n-2}y'(0) - \\cdots - y^{(n-1)}(0)

Solving First-Order IVPs

Let's solve a first-order linear IVP step by step using the Laplace transform method.

Example: Exponential Decay

Problem: Solve $y' + 2y = 0$ with $y(0) = 3$

Step 1: Take the Laplace transform of both sides.

Transform of $y'$ : $sY(s) - y(0) = sY(s) - 3$

Transform of $2y$ : $2Y(s)$

Transform of $0$ : $0$

Step 2: Write the algebraic equation.

sY(s) - 3 + 2Y(s) = 0

Step 3: Solve for Y(s).

$(s + 2)Y(s) = 3$

$Y(s) = \\frac{3}{s + 2}$

Step 4: Apply the inverse Laplace transform.

From our table of transforms: $\\mathcal{L}^{-1}\\left\\{\\frac{1}{s+a}\\right\\} = e^{-at}$

Solution:

y(t) = 3e^{-2t}

Verification: Check that $y(0) = 3$ ✓ and $y' + 2y = -6e^{-2t} + 6e^{-2t} = 0$ ✓

Example: First-Order with Forcing

Problem: Solve $y' + y = 6$ with $y(0) = 0$

Step 1-2: Transform both sides:

$sY(s) - 0 + Y(s) = \\frac{6}{s}$

$(s + 1)Y(s) = \\frac{6}{s}$

Step 3: Solve for Y(s):

Y(s) = \\frac{6}{s(s+1)}

Step 4: Use partial fractions:

$\\frac{6}{s(s+1)} = \\frac{A}{s} + \\frac{B}{s+1}$

Solving: $A = 6, B = -6$

$Y(s) = \\frac{6}{s} - \\frac{6}{s+1}$

Inverse transform:

Solution:

y(t) = 6 - 6e^{-t} = 6(1 - e^{-t})

This represents a quantity approaching the steady state $y = 6$ exponentially.

Interactive: IVP Solver

Explore the step-by-step Laplace transform solution for various initial value problems:

Laplace Transform IVP Solver

Select Initial Value Problem:

Differential Equation

y' + 2y = 0

Initial Conditions

y(0) = 3

Solution Process

Step 1: Take the Laplace Transform

Apply the Laplace transform to both sides of the differential equation, using the derivative property:

\mathcal{L}\{y'\} = sY(s) - y(0)

\mathcal{L}\{y''\} = s^2Y(s) - sy(0) - y'(0)

Solution Graph

Physical Interpretation: Models radioactive decay, RC circuit discharge, Newton's cooling

Solving Second-Order IVPs

Second-order equations are where the Laplace method truly shines. The classical approach requires finding roots of characteristic equations, handling complex numbers, and determining two constants. The Laplace method handles all this algebraically.

Example: Simple Harmonic Oscillator

Problem: Solve $y'' + 4y = 0$ with $y(0) = 1, y'(0) = 0$

Step 1: Apply the Laplace transform:

$\\mathcal{L}\\{y''\\} = s^2Y(s) - sy(0) - y'(0) = s^2Y(s) - s - 0$

$\\mathcal{L}\\{4y\\} = 4Y(s)$

Step 2: Algebraic equation:

s^2Y(s) - s + 4Y(s) = 0

Step 3: Solve for Y(s):

$(s^2 + 4)Y(s) = s$

$Y(s) = \\frac{s}{s^2 + 4}$

Step 4: Recognize the transform pair: $\\mathcal{L}\\{\\cos(\\omega t)\\} = \\frac{s}{s^2 + \\omega^2}$

Here $\\omega^2 = 4$ , so $\\omega = 2$ :

Solution:

y(t) = \\cos(2t)

Example: Damped Oscillator

Problem: Solve $y'' + 2y' + 5y = 0$ with $y(0) = 1, y'(0) = 0$

Step 1-2: Transform:

$s^2Y(s) - s - 0 + 2[sY(s) - 1] + 5Y(s) = 0$

$(s^2 + 2s + 5)Y(s) = s + 2$

Step 3: Complete the square in the denominator:

$s^2 + 2s + 5 = (s + 1)^2 + 4$

$Y(s) = \\frac{s + 2}{(s+1)^2 + 4}$

Step 4: Rewrite for inverse transform:

$Y(s) = \\frac{(s+1)}{(s+1)^2 + 4} + \\frac{1}{(s+1)^2 + 4}$

$= \\frac{(s+1)}{(s+1)^2 + 4} + \\frac{1}{2} \\cdot \\frac{2}{(s+1)^2 + 4}$

Using the shift theorem $\\mathcal{L}^{-1}\\{F(s+a)\\} = e^{-at}\\mathcal{L}^{-1}\\{F(s)\\}$ :

Solution:

y(t) = e^{-t}\\left(\\cos(2t) + \\frac{1}{2}\\sin(2t)\\right)

This is a decaying oscillation — the characteristic behavior of a damped system.

Time Domain vs s-Domain: A Comparison

Let's compare the two approaches side by side to see why engineers prefer the Laplace method:

Time Domain vs Laplace Domain Methods

Classical Time Domain Method

1Write characteristic equation

2Find roots (may need quadratic formula)

3Write general solution

4Apply initial conditions

5Solve system of equations for constants

Challenges: Finding roots can be tedious, especially for complex roots. Initial conditions applied at the end.

Laplace Transform Method

1Take Laplace transform of both sides

2Substitute initial conditions directly

3Solve algebra for Y(s)

4Apply inverse Laplace transform

Advantages: Initial conditions built in from the start. Differentiation becomes multiplication by s. Purely algebraic process.

The Big Picture

The Laplace transform converts calculus problems (differentiation, integration) into algebra problems (multiplication, division). Initial conditions are incorporated automatically through the derivative property, eliminating the need to solve for arbitrary constants at the end.

The Laplace Transform Roadmap

Time Domain

y'' + y' + y = f(t)

Differential Equation

s-Domain

(s^2 + s + 1)Y(s) = ...

Algebraic Equation

Time Domain

y(t) = ...

Solution

Aspect	Classical Method	Laplace Method
Starting point	Find homogeneous + particular solutions	Transform entire equation
Initial conditions	Applied at the end	Built in from the start
Differentiation	Calculus operations	Multiplication by s
Finding constants	Solve system of equations	Automatic from ICs
Complex roots	Must handle explicitly	Completing the square suffices
Forcing functions	Method of undetermined coefficients	Just transform and algebra

Applications: Circuit Analysis

Electrical circuits are perhaps the most natural application of Laplace transforms for IVPs. The circuit elements have simple Laplace domain representations:

Element	Time Domain	Laplace Domain
Resistor R	v = Ri	V(s) = RI(s)
Inductor L	v = L(di/dt)	V(s) = sLI(s) - Li(0)
Capacitor C	i = C(dv/dt)	I(s) = sCV(s) - Cv(0)

RC Circuit Transient

Consider an RC circuit with initial capacitor voltage $v_0$ . The governing equation is:

RC\\frac{dv}{dt} + v = V_{in}

Taking the Laplace transform with initial condition $v(0) = v_0$ :

$RC[sV(s) - v_0] + V(s) = \\frac{V_{in}}{s}$

$(RCs + 1)V(s) = \\frac{V_{in}}{s} + RCv_0$

$V(s) = \\frac{V_{in}}{s(RCs + 1)} + \\frac{RCv_0}{RCs + 1}$

After partial fractions and inverse transform:

v(t) = V_{in}(1 - e^{-t/RC}) + v_0 e^{-t/RC}

The time constant $\\tau = RC$ determines how fast the circuit responds. After $5\\tau$ , the transient has decayed to less than 1% of its initial value.

Interactive: RC Circuit Transients

Explore how an RC circuit charges and discharges. Adjust the resistance, capacitance, and initial conditions:

RC Circuit Transient Analysis via Laplace Transform

DE:

RC\frac{dv}{dt} + v = V_{in}

Laplace:

RC(sV(s) - v_0) + V(s) = \frac{V_{in}}{s}

Time constant:

\tau = RC = 1000.0\text{ ms}

Resistance R = 1000 Ω

Capacitance C = 1.00 mF

Initial Voltage v(0) = 5 V

Laplace Transform Solution

v(t) = V_{in}(1 - e^{-t/\tau}) + v_0 e^{-t/\tau} = 10(1 - e^{-t/1000.0\text{ms}}) + 5e^{-t/1000.0\text{ms}}

Applications: Mechanical Systems

The same mathematics governs mechanical systems. A mass-spring-damper system obeys:

m\\frac{d^2x}{dt^2} + c\\frac{dx}{dt} + kx = F(t)

where $m$ is mass, $c$ is damping coefficient, $k$ is spring constant, and $F(t)$ is external force.

Free Vibration

With $F(t) = 0$ , initial displacement $x_0$ , and initial velocity $v_0$ :

ms^2X(s) - msx_0 - mv_0 + csX(s) - cx_0 + kX(s) = 0

Solving for $X(s)$ :

X(s) = \\frac{(ms + c)x_0 + mv_0}{ms^2 + cs + k}

The character of the solution depends on the discriminant $c^2 - 4mk$ :

Overdamped ( $c^2 > 4mk$ ): Two real roots, exponential decay without oscillation
Critically damped ( $c^2 = 4mk$ ): Repeated root, fastest non-oscillatory decay
Underdamped ( $c^2 < 4mk$ ): Complex roots, decaying oscillation

The Same Math

The RC circuit equation $RC\\frac{dv}{dt} + v = V_{in}$ is mathematically identical to a first-order mechanical system. The mass-spring-damper is analogous to an RLC circuit. This is why electrical engineers use mechanical analogies and vice versa.

Machine Learning Connections

The mathematics of IVPs appears throughout machine learning, often in surprising ways:

Neural ODEs

Neural Ordinary Differential Equations (Neural ODEs) are a deep learning architecture where the hidden state evolves according to a learned ODE:

\\frac{d\\mathbf{h}}{dt} = f(\\mathbf{h}, t, \\theta)

The forward pass literally solves an IVP! Given input $\\mathbf{h}(0)$ , we integrate forward to get $\\mathbf{h}(T)$ . Backpropagation uses the adjoint sensitivity method, which also involves solving IVPs.

Momentum in Optimization

Heavy ball momentum in gradient descent can be written as:

m\\ddot{x} + \\gamma\\dot{x} = -\\nabla f(x)

This is exactly the damped oscillator equation! The underdamped case ( $\\gamma^2 < 4m$ ) explains why loss curves often oscillate during training.

Exponential Learning Rate Decay

The exponential decay schedule $\\eta(t) = \\eta_0 e^{-\\lambda t}$ is the solution to:

\\frac{d\\eta}{dt} = -\\lambda \\eta, \\quad \\eta(0) = \\eta_0

This is the simplest first-order IVP, the same equation governing radioactive decay and RC circuit discharge.

Control Systems in Reinforcement Learning

In continuous control tasks, the environment dynamics are often modeled as differential equations. The policy network outputs actions that drive the system:

\\dot{\\mathbf{x}} = f(\\mathbf{x}, \\mathbf{u}, t)

where $\\mathbf{u} = \\pi(\\mathbf{x}; \\theta)$ is the policy output. Understanding IVP solutions helps in designing stable controllers and analyzing system behavior.

Python Implementation

Solving IVPs with Laplace Transforms

Here's how to solve initial value problems using SymPy's symbolic Laplace transform capabilities:

Laplace Transform IVP Solver

🐍laplace_ivp_solver.py

Explanation(6)

Code(126)

10Symbolic Math Setup

SymPy allows us to work symbolically with Laplace transforms, performing exact algebraic manipulations rather than numerical approximations.

25First-Order IVP

For y' + 2y = 0 with y(0) = 3, we apply the derivative property: L{y'} = sY(s) - y(0). The initial condition enters directly into the algebraic equation.

37Inverse Transform

After solving for Y(s) = 3/(s+2), we use inverse_laplace_transform to get y(t) = 3e^(-2t). The transform pair L{e^(-at)} = 1/(s+a) is fundamental here.

52Second-Order IVP

For y'' + 4y = 0, we need L{y''} = s²Y(s) - sy(0) - y'(0). Both initial conditions appear in the transformed equation, making them easy to incorporate.

68Damped Oscillator

The damped case y'' + 2y' + 5y = 0 leads to complex roots. The denominator s² + 2s + 5 = (s+1)² + 4 gives exponentially decaying oscillations.

88Numerical Verification

Comparing our Laplace transform solution with scipy's odeint confirms the algebraic method gives the same result as numerical integration.

120 lines without explanation

1import numpy as np
2import sympy as sp
3from sympy import symbols, Function, dsolve, laplace_transform
4from sympy import inverse_laplace_transform, exp, cos, sin
5from scipy.integrate import odeint
6import matplotlib.pyplot as plt
7
8def solve_ivp_laplace():
9    """
10    Solve initial value problems using Laplace transforms.
11    This demonstrates the algebraic approach to differential equations.
12    """
13    # Define symbols
14    t, s = symbols('t s', positive=True, real=True)
15
16    # ============================================
17    # Example 1: First-Order IVP
18    # y' + 2y = 0, y(0) = 3
19    # ============================================
20    print("=" * 50)
21    print("Example 1: y' + 2y = 0, y(0) = 3")
22    print("=" * 50)
23
24    # Step 1: Apply Laplace transform
25    # L{y'} = sY(s) - y(0) = sY(s) - 3
26    # L{2y} = 2Y(s)
27    # Transformed equation: sY(s) - 3 + 2Y(s) = 0
28
29    Y = symbols('Y')  # Y(s)
30    y0 = 3  # Initial condition
31
32    # Step 2: Solve algebraically for Y(s)
33    # (s + 2)Y(s) = 3
34    # Y(s) = 3/(s + 2)
35    Y_s = 3 / (s + 2)
36    print(f"Step 1-2: Y(s) = {Y_s}")
37
38    # Step 3: Inverse Laplace transform
39    y_t = inverse_laplace_transform(Y_s, s, t)
40    print(f"Step 3: y(t) = {y_t}")
41
42    # Verify: y(0) should equal 3
43    y_at_0 = y_t.subs(t, 0)
44    print(f"Verification: y(0) = {y_at_0}")
45
46    # ============================================
47    # Example 2: Second-Order Oscillator
48    # y'' + 4y = 0, y(0) = 1, y'(0) = 0
49    # ============================================
50    print("\n" + "=" * 50)
51    print("Example 2: y'' + 4y = 0, y(0) = 1, y'(0) = 0")
52    print("=" * 50)
53
54    # Transform: s²Y(s) - s·y(0) - y'(0) + 4Y(s) = 0
55    # s²Y(s) - s + 4Y(s) = 0
56    # (s² + 4)Y(s) = s
57    Y_s_2 = s / (s**2 + 4)
58    print(f"Y(s) = {Y_s_2}")
59
60    y_t_2 = inverse_laplace_transform(Y_s_2, s, t)
61    print(f"y(t) = {y_t_2}")
62
63    # ============================================
64    # Example 3: Damped Oscillator
65    # y'' + 2y' + 5y = 0, y(0) = 1, y'(0) = 0
66    # ============================================
67    print("\n" + "=" * 50)
68    print("Example 3: y'' + 2y' + 5y = 0, y(0) = 1, y'(0) = 0")
69    print("=" * 50)
70
71    # Transform: s²Y - s + 2(sY - 1) + 5Y = 0
72    # (s² + 2s + 5)Y = s + 2
73    Y_s_3 = (s + 2) / (s**2 + 2*s + 5)
74
75    # Complete the square: s² + 2s + 5 = (s+1)² + 4
76    # Rewrite: Y(s) = (s + 2)/((s+1)² + 4)
77    #                = (s+1)/((s+1)² + 4) + 1/((s+1)² + 4)
78    #                = (s+1)/((s+1)² + 4) + (1/2)·(2/((s+1)² + 4))
79
80    y_t_3 = inverse_laplace_transform(Y_s_3, s, t)
81    print(f"Y(s) = {Y_s_3}")
82    print(f"y(t) = {y_t_3}")
83
84    return y_t, y_t_2, y_t_3
85
86def numerical_comparison():
87    """
88    Compare Laplace transform solution with numerical ODE solver.
89    """
90    # System: y' + 2y = 0, y(0) = 3
91    # Analytical: y(t) = 3e^(-2t)
92
93    def ode_system(y, t):
94        return -2 * y
95
96    t_span = np.linspace(0, 4, 100)
97    y0 = [3.0]
98
99    # Numerical solution
100    y_numerical = odeint(ode_system, y0, t_span)
101
102    # Analytical (from Laplace)
103    y_analytical = 3 * np.exp(-2 * t_span)
104
105    # Plot comparison
106    plt.figure(figsize=(10, 6))
107    plt.plot(t_span, y_numerical, 'b-', linewidth=2,
108             label='Numerical (odeint)')
109    plt.plot(t_span, y_analytical, 'r--', linewidth=2,
110             label='Laplace Transform Solution')
111    plt.xlabel('Time t')
112    plt.ylabel('y(t)')
113    plt.title("IVP: y' + 2y = 0, y(0) = 3")
114    plt.legend()
115    plt.grid(True, alpha=0.3)
116    plt.savefig('ivp_comparison.png', dpi=150)
117    print("\nPlot saved to ivp_comparison.png")
118
119    # Compute error
120    max_error = np.max(np.abs(y_numerical.flatten() - y_analytical))
121    print(f"Maximum error: {max_error:.2e}")
122
123# Run demonstrations
124if __name__ == "__main__":
125    solve_ivp_laplace()
126    numerical_comparison()

Laplace Transforms in Machine Learning

Here's how IVP solutions connect to machine learning concepts:

Laplace Transforms in ML

🐍laplace_ml_connections.py

Explanation(5)

Code(159)

20Transfer Functions

PID controllers are naturally expressed in the Laplace domain. The transfer function G(s) relates input to output, and step response is found by solving an IVP.

45Neural ODEs

Neural ODEs parameterize dynamics dy/dt = f(y,t,θ) with a neural network. The forward pass literally solves an IVP! Backprop uses the adjoint sensitivity method.

73Exponential Learning Rate

Exponential decay η(t) = η₀e^(-λt) is the solution to dη/dt = -λη, the simplest first-order IVP. This connection to differential equations appears throughout ML.

93RNN as Continuous Dynamics

An RNN's hidden state update can be viewed as Euler discretization of an ODE. In Laplace terms, the RNN is a low-pass filter smoothing the input sequence.

115Momentum as Second-Order ODE

Heavy ball momentum x'' + γx' + ∇²f·x = 0 is exactly the damped oscillator equation. Underdamped systems oscillate during convergence, which we observe in training loss curves.

154 lines without explanation

1import numpy as np
2import torch
3import torch.nn as nn
4from scipy import signal
5from scipy.integrate import odeint
6
7def laplace_in_machine_learning():
8    """
9    Laplace transforms and IVP solutions appear in ML contexts:
10    1. Control systems for RL
11    2. Neural ODEs
12    3. Continuous-time models
13    4. Signal processing in audio/time-series
14    """
15
16    # ============================================
17    # 1. Transfer Functions in Control Theory
18    # ============================================
19    print("=" * 50)
20    print("1. Transfer Functions for Continuous Control")
21    print("=" * 50)
22
23    # A PID controller in the Laplace domain:
24    # G(s) = Kp + Ki/s + Kd·s
25    # This is the transfer function from error to control signal
26
27    Kp, Ki, Kd = 1.0, 0.5, 0.1  # PID gains
28
29    # Define transfer function numerator and denominator
30    # G(s) = (Kd·s² + Kp·s + Ki) / s
31    num = [Kd, Kp, Ki]
32    den = [1, 0]  # Just 's' in denominator
33
34    # Create transfer function
35    pid_tf = signal.TransferFunction(num, den)
36    print(f"PID Transfer Function: ({Kd}s² + {Kp}s + {Ki}) / s")
37
38    # Step response (equivalent to solving IVP with step input)
39    t, response = signal.step(pid_tf)
40    print(f"Step response computed for t in [0, {t[-1]:.2f}]")
41
42    # ============================================
43    # 2. Neural ODE Concept
44    # ============================================
45    print("\n" + "=" * 50)
46    print("2. Neural ODE: Learning the Dynamics")
47    print("=" * 50)
48
49    # In Neural ODEs, we learn f(y, t, θ) where dy/dt = f(y, t, θ)
50    # The IVP solution y(t) = y(0) + ∫₀ᵗ f(y(τ), τ, θ) dτ
51    # is computed via numerical integration (like solving IVP)
52
53    class SimpleNeuralODE(nn.Module):
54        """A minimal Neural ODE dynamics model."""
55        def __init__(self, dim=2, hidden=32):
56            super().__init__()
57            self.net = nn.Sequential(
58                nn.Linear(dim, hidden),
59                nn.Tanh(),
60                nn.Linear(hidden, hidden),
61                nn.Tanh(),
62                nn.Linear(hidden, dim)
63            )
64
65        def forward(self, t, y):
66            """Compute dy/dt = f(y, t; θ)"""
67            return self.net(y)
68
69    # Initialize model
70    neural_ode = SimpleNeuralODE(dim=2)
71    print(f"Neural ODE model: {sum(p.numel() for p in neural_ode.parameters())} parameters")
72
73    # The forward pass solves an IVP!
74    y0 = torch.tensor([[1.0, 0.0]])
75
76    # In practice, we'd use torchdiffeq.odeint_adjoint
77    # dy_dt = neural_ode(0, y0)
78    print("Neural ODE computes dy/dt, IVP solver integrates")
79
80    # ============================================
81    # 3. Exponential Decay in Learning Rates
82    # ============================================
83    print("\n" + "=" * 50)
84    print("3. Exponential Decay (IVP Solution)")
85    print("=" * 50)
86
87    # Learning rate schedule: η(t) = η₀ · e^(-λt)
88    # This is the solution to dη/dt = -λη, η(0) = η₀
89    # (Same form as RC circuit discharge!)
90
91    eta_0 = 0.1  # Initial learning rate
92    decay_rate = 0.01  # λ
93
94    steps = np.arange(0, 1000)
95    lr_schedule = eta_0 * np.exp(-decay_rate * steps)
96
97    print(f"Initial LR: {eta_0}")
98    print(f"LR at step 100: {lr_schedule[100]:.4f}")
99    print(f"LR at step 500: {lr_schedule[500]:.4f}")
100
101    # ============================================
102    # 4. RNN as Continuous Dynamics
103    # ============================================
104    print("\n" + "=" * 50)
105    print("4. RNN Hidden State as ODE")
106    print("=" * 50)
107
108    # Standard RNN: h_{t+1} = tanh(W_h·h_t + W_x·x_t + b)
109    # Continuous-time version: dh/dt = -h + tanh(W_h·h + W_x·x + b)
110    # This is an IVP! The discrete RNN is Euler's method.
111
112    # For a linear system: dh/dt = -αh + x(t)
113    # Laplace: sH(s) - h(0) = -αH(s) + X(s)
114    # (s + α)H(s) = h(0) + X(s)
115    # H(s) = h(0)/(s+α) + X(s)/(s+α)
116
117    # The transfer function 1/(s+α) is a low-pass filter!
118    alpha = 1.0
119    num_rnn = [1]
120    den_rnn = [1, alpha]
121
122    rnn_tf = signal.TransferFunction(num_rnn, den_rnn)
123    w, H = signal.freqresp(rnn_tf)
124
125    print(f"RNN as low-pass filter with time constant τ = 1/α = {1/alpha}")
126    print(f"At ω = 1: |H(jω)| = {np.abs(H[np.argmin(np.abs(w-1))]):.3f}")
127
128    # ============================================
129    # 5. Momentum in Optimization
130    # ============================================
131    print("\n" + "=" * 50)
132    print("5. Momentum as Second-Order IVP")
133    print("=" * 50)
134
135    # Heavy ball momentum: m·v' + γ·v = -∇f(x)
136    # This is a second-order ODE in x: m·x'' + γ·x' = -∇f(x)
137    # Analogous to damped harmonic oscillator!
138
139    # For quadratic f(x) = (1/2)x², the system becomes:
140    # x'' + (γ/m)x' + (1/m)x = 0
141    # Characteristic equation: s² + (γ/m)s + 1/m = 0
142
143    m, gamma = 1.0, 0.5  # Mass and friction
144
145    # Roots determine convergence behavior
146    discriminant = (gamma/m)**2 - 4/m
147    print(f"System: x'' + {gamma/m:.2f}x' + {1/m:.2f}x = 0")
148    print(f"Discriminant: {discriminant:.4f}")
149
150    if discriminant < 0:
151        print("Underdamped: Oscillatory convergence (like training!)")
152    elif discriminant == 0:
153        print("Critically damped: Fastest non-oscillatory convergence")
154    else:
155        print("Overdamped: Slow monotonic convergence")
156
157# Run ML demonstrations
158if __name__ == "__main__":
159    laplace_in_machine_learning()

Common Mistakes to Avoid

Mistake 1: Forgetting Initial Conditions in the Transform

Wrong: $\\mathcal{L}\\{y'\\} = sY(s)$

Correct: $\\mathcal{L}\\{y'\\} = sY(s) - y(0)$

The initial condition term is essential. This is what makes the Laplace method work for IVPs!

Mistake 2: Incorrect Transform for Second Derivative

Wrong: $\\mathcal{L}\\{y''\\} = s^2Y(s) - y(0)$

Correct: $\\mathcal{L}\\{y''\\} = s^2Y(s) - sy(0) - y'(0)$

Both initial conditions appear. Note the factors of $s$ on $y(0)$ .

Mistake 3: Partial Fractions Errors

When decomposing $Y(s)$ , be careful with signs and coefficients. Always verify by recombining the fractions.

For example, $\\frac{6}{s(s+1)} = \\frac{6}{s} - \\frac{6}{s+1}$ , not $\\frac{3}{s} + \\frac{3}{s+1}$ .

Mistake 4: Forgetting the Shift Theorem

When the denominator is $(s+a)^2 + b^2$ , you need the shift theorem:

$\\mathcal{L}^{-1}\\left\\{\\frac{s+a}{(s+a)^2 + b^2}\\right\\} = e^{-at}\\cos(bt)$

$\\mathcal{L}^{-1}\\left\\{\\frac{b}{(s+a)^2 + b^2}\\right\\} = e^{-at}\\sin(bt)$

Mistake 5: Wrong Sign for Trigonometric Transforms

Remember:

$\\mathcal{L}\\{\\cos(\\omega t)\\} = \\frac{s}{s^2 + \\omega^2}$ (has $s$ in numerator)

$\\mathcal{L}\\{\\sin(\\omega t)\\} = \\frac{\\omega}{s^2 + \\omega^2}$ (has $\\omega$ in numerator)

Test Your Understanding

Quiz: Solving IVPs with Laplace Transforms

Question 1 of 8Score: 0/0

When taking the Laplace transform of y'(t), the result includes which term involving the initial condition?

Summary

The Laplace transform provides an elegant algebraic approach to solving initial value problems. By converting differentiation into multiplication and incorporating initial conditions automatically, it simplifies what would otherwise be tedious calculus.

Key Formulas

Transform	Formula	Application
First Derivative	L{y'} = sY(s) - y(0)	Incorporates initial position/value
Second Derivative	L{y''} = s²Y(s) - sy(0) - y'(0)	Incorporates position and velocity
Shift Theorem	L^(-1){F(s+a)} = e^(-at)L^(-1){F(s)}	Handles damped oscillations
Cosine	L{cos(ωt)} = s/(s² + ω²)	Undamped oscillations
Sine	L{sin(ωt)} = ω/(s² + ω²)	Undamped oscillations

Key Takeaways

Calculus becomes algebra: Differentiation becomes multiplication by $s$ , making IVPs algebraic problems.
Initial conditions are automatic: They appear naturally in the derivative transforms, eliminating the need to solve for constants at the end.
The method is systematic: Transform → Algebra → Inverse Transform. No guessing required.
Partial fractions are essential: Most $Y(s)$ expressions need decomposition before inverse transforming.
Physical interpretation guides understanding: RC circuits, mass-spring systems, and ML dynamics all follow the same mathematics.
ML connections are deep: From neural ODEs to momentum optimization, IVP solutions appear throughout machine learning.

The Core Insight:

"The Laplace transform converts the hard problem of solving differential equations into the easier problem of solving algebra. Initial conditions come along for free."

Coming Next: In Step Functions and Discontinuous Forcing, we'll learn how Laplace transforms elegantly handle sudden changes in input — from switching circuits to impulsive forces — using the Heaviside step function and shifting theorems.