Boo-AI — Master Artificial Intelligence by Building from Scratch

Learning Objectives

By the end of this section, you will be able to:

Understand the Heaviside step function $u(t-a)$ and its role in modeling discontinuous phenomena
Compute the Laplace transform of step functions and their products with other functions
Apply the Second Shifting Theorem (time-shift property) to handle delayed signals and discontinuities
Express piecewise-defined functions using combinations of step functions
Solve differential equations with discontinuous forcing functions using Laplace transforms
Connect step functions to their widespread applications in engineering, control systems, and machine learning

The Big Picture: Modeling the Discontinuous World

"The real world is full of sudden changes—switches turning on, forces suddenly applied, signals beginning. Step functions give us the mathematical language to describe these discontinuities with precision and elegance."

Until now, we have primarily worked with smooth, continuous functions. But reality is rarely so smooth. Consider these everyday scenarios:

A light switch is flipped ON—the voltage jumps instantly from 0 to 120V
A rocket engine ignites—thrust goes from 0 to maximum in milliseconds
A drug is administered—concentration jumps from 0 to some initial dose
A training curriculum changes—learning rate drops by a factor of 10

These scenarios share a common feature: a quantity changes abruptly at a specific time. To model such phenomena, we need a mathematical object that can represent "off" before a certain time and "on" after it. This is precisely what the Heaviside step function provides.

Why Step Functions Matter

The step function is the bridge between the idealized world of continuous mathematics and the discontinuous reality of engineering and physics. Combined with Laplace transforms, it becomes a powerful tool for solving differential equations that describe systems subjected to sudden changes.

The combination of step functions with Laplace transforms gives us a systematic method to:

Express any piecewise function compactly
Transform discontinuous forcing functions into the s-domain
Solve ODEs with sudden inputs algebraically
Find solutions that automatically account for discontinuities

Historical Context: Oliver Heaviside's Revolutionary Function

The step function is named after Oliver Heaviside (1850–1925), a self-taught English electrical engineer and mathematician who made profound contributions to the field of electrical circuit theory and mathematical physics.

Heaviside worked as a telegraph operator and became fascinated by the mathematical problems of signal transmission. He developed what we now call operational calculus—a precursor to modern Laplace transform methods—to solve differential equations arising in circuit analysis.

Heaviside's Bold Approach

Heaviside's methods were considered controversial during his lifetime. He treated the differential operator $D = d/dt$ as if it were an algebraic quantity, writing solutions that seemed to come from nowhere. His critics demanded rigorous justification, but Heaviside famously replied:

"Shall I refuse my dinner because I do not fully understand the process of digestion?" — Oliver Heaviside

The rigorous foundation for Heaviside's methods came later through the Laplace transform, which provides the mathematical justification for his operational techniques. The step function now bears his name as a tribute to his pioneering work on systems with sudden inputs.

Heaviside's Other Contributions

Besides the step function, Heaviside reformulated Maxwell's equations into the compact vector notation we use today, developed the theory of transmission lines, and co-predicted the existence of the ionosphere (sometimes called the "Heaviside layer").

The Heaviside Step Function

The Heaviside step function (also called the unit step function) is defined as:

Definition: Heaviside Step Function

u(t) = \begin{cases} 0 & t < 0 \\ 1 & t \geq 0 \end{cases}

The function "turns on" at t = 0, jumping from 0 to 1.

The step function represents the ideal mathematical model of a switch: "off" (value 0) before time 0, and "on" (value 1) at and after time 0.

Shifted Step Function

In most applications, we want the "switch" to turn on at some time $t = a$ other than zero. The shifted step function is:

Shifted Step Function

u(t-a) = \begin{cases} 0 & t < a \\ 1 & t \geq a \end{cases}

The function "turns on" at time t = a.

The notation $u(t-a)$ shifts the "switch on" time from 0 to $a$ . This is a horizontal shift to the right by $a$ units.

Key Properties

Property	Expression	Interpretation
Turn-on at a	u(t-a)	0 for t < a, 1 for t ≥ a
Turn-off at b	1 - u(t-b)	1 for t < b, 0 for t ≥ b
Rectangular pulse	u(t-a) - u(t-b)	1 for a ≤ t < b, 0 elsewhere
Scaling signal	A·u(t-a)	Jump to amplitude A at t = a
Product rule	f(t)·u(t-a)	f(t) for t ≥ a, 0 for t < a

Interactive: Step Function Explorer

Explore how the step function works by adjusting the shift parameter $a$ . Observe how combinations of step functions create pulses and other patterns:

📈Step Function Explorer

Shift a = 2.0Step turns on at t = 2.0

Amplitude = 1.0Height of step

Step Function Expression:

f(t) = 1.0 \cdot u(t - 2.0)

Laplace Transform:

\mathcal{L}\{f(t)\} = \frac{1.0 \cdot e^{-2.0s}}{s}

Laplace Transform of the Step Function

To use step functions with the Laplace transform method, we need to know their transforms. Let's derive them.

Transform of u(t)

The Laplace transform of the basic step function $u(t)$ is straightforward:

Derivation:

$\mathcal{L}\{u(t)\} = \int_0^{\infty} u(t) e^{-st} \, dt$

Since $u(t) = 1$ for $t \geq 0$ :

$= \int_0^{\infty} e^{-st} \, dt = \left[ -\frac{1}{s} e^{-st} \right]_0^{\infty}$

$= 0 - \left( -\frac{1}{s} \right) = \frac{1}{s}$

Laplace Transform of Unit Step

\mathcal{L}\{u(t)\} = \frac{1}{s}

Transform of Shifted Step u(t-a)

For the shifted step function with $a > 0$ :

Derivation:

$\mathcal{L}\{u(t-a)\} = \int_0^{\infty} u(t-a) e^{-st} \, dt$

Since $u(t-a) = 0$ for $t < a$ and $u(t-a) = 1$ for $t \geq a$ :

$= \int_a^{\infty} e^{-st} \, dt = \left[ -\frac{1}{s} e^{-st} \right]_a^{\infty}$

$= 0 - \left( -\frac{1}{s} e^{-sa} \right) = \frac{e^{-as}}{s}$

Laplace Transform of Shifted Step

\mathcal{L}\{u(t-a)\} = \frac{e^{-as}}{s} \quad (a > 0)

The Exponential Factor

The factor $e^{-as}$ encodes the time delay of $a$ units in the s-domain. Whenever you see $e^{-as}$ in a Laplace transform, it signals a time shift of $a$ in the time domain.

The Second Shifting Theorem

The Second Shifting Theorem (also called the Time-Shift Property) is one of the most important results for handling discontinuous functions. It tells us how to transform a function that "turns on" at time $a$ .

The Second Shifting Theorem

\mathcal{L}\{f(t)\} = F(s)

, then:

\mathcal{L}\{f(t-a) \cdot u(t-a)\} = e^{-as} F(s)

A time shift by

a

in the time domain corresponds to multiplication by

e^{-as}

in the s-domain.

Understanding the Theorem

The expression $f(t-a) \cdot u(t-a)$ represents:

For $t < a$ : the value is 0 (the step function is off)
For $t \geq a$ : the value is $f(t-a)$ (the original function, but shifted right by $a$ )

This is exactly what we need to model a function that "turns on" at time $a$ .

Inverse Second Shifting Theorem

Equally important is the inverse direction: given a transform with an exponential factor, we can find the time-domain function:

Inverse Second Shifting Theorem

\mathcal{L}^{-1}\{F(s)\} = f(t)

, then:

\mathcal{L}^{-1}\{e^{-as} F(s)\} = f(t-a) \cdot u(t-a)

Examples

Time Function	Laplace Transform	Notes
u(t-3)	e^(-3s)/s	Step turning on at t=3
(t-2)·u(t-2)	e^(-2s)/s²	Ramp starting at t=2
e^(-(t-5))·u(t-5)	e^(-5s)/(s+1)	Exponential decay starting at t=5
sin(t-π)·u(t-π)	e^(-πs)/(s²+1)	Sine wave starting at t=π
(t-4)²·u(t-4)	2e^(-4s)/s³	Parabola starting at t=4

Interactive: Second Shifting Theorem

Visualize how the Second Shifting Theorem works. Select a function and shift parameter to see the relationship between the time domain and s-domain:

⏰Second Shifting Theorem Visualizer

Time Shift a = 2.0Function turns on at t = 2.0

Time Domain:

f(t-2.0) \cdot u(t-2.0)

where

f(t) = t

s-Domain (Laplace Transform):

\frac{e^{-2.0s}}{s^2}

= e^{-2.0s} \cdot F(s)

Second Shifting Theorem:

\mathcal{L}\{f(t-a) \cdot u(t-a)\} = e^{-as} \cdot F(s)

Time shift by

a

→ Multiply by

e^{-as}

in s-domain

Writing Piecewise Functions Using Step Functions

One of the most powerful applications of step functions is expressing piecewise-defined functions in a compact form. This allows us to take their Laplace transforms directly.

The General Strategy

To convert a piecewise function to step function form:

Identify the breakpoints where the function changes definition
Express each piece in terms of step functions that turn on and off at appropriate times
Combine using addition and subtraction

Example 1: Simple Two-Piece Function

Convert: $f(t) = \begin{cases} 0 & 0 \leq t < 2 \\ 5 & t \geq 2 \end{cases}$

Solution:

The function is 0 until $t = 2$ , then jumps to 5. This is simply:

f(t) = 5 \cdot u(t-2)

Example 2: Three-Piece Function

Convert: $g(t) = \begin{cases} 0 & 0 \leq t < 1 \\ t & 1 \leq t < 3 \\ 4 & t \geq 3 \end{cases}$

Solution:

Break this into pieces using "turn on" and "turn off" logic:

For $1 \leq t < 3$ : the function is $t$ , which turns on at $t=1$ and off at $t=3$
For $t \geq 3$ : the function is 4, which turns on at $t=3$

The ramp $t$ from 1 to 3 needs special handling. We write:

g(t) = t \cdot [u(t-1) - u(t-3)] + 4 \cdot u(t-3)

Expanding and simplifying:

g(t) = t \cdot u(t-1) - t \cdot u(t-3) + 4 \cdot u(t-3)

Or equivalently:

g(t) = t \cdot u(t-1) + (4-t) \cdot u(t-3)

Example 3: Rectangular Pulse

Convert: A pulse of height $A$ from $t = a$ to $t = b$ :

$f(t) = \begin{cases} 0 & t < a \\ A & a \leq t < b \\ 0 & t \geq b \end{cases}$

Solution:

f(t) = A \cdot [u(t-a) - u(t-b)]

The Laplace transform is:

\mathcal{L}\{f(t)\} = A \cdot \left( \frac{e^{-as} - e^{-bs}}{s} \right)

The Rewriting Trick

When using the Second Shifting Theorem, you must express the function as $f(t-a) \cdot u(t-a)$ —the argument of $f$ must be $(t-a)$ , matching the step function. For example, to transform $t^2 \cdot u(t-3)$ , rewrite as $[(t-3)+3]^2 \cdot u(t-3)$ and expand.

Interactive: Piecewise Function Builder

Build piecewise functions visually and see their step function representations and Laplace transforms:

🔧Piecewise Function Builder

Adjust piece values (click to select):

0 ≤ t < 2

Value:0

2 ≤ t < 4

Value:3

4 ≤ t < 8

Value:1

Step Function Representation:

f(t) = + 3 \cdot u(t-2) - 2 \cdot u(t-4)

Piecewise Definition:

f(t) = \begin{cases} 0 & 0 \leq t < 2 \\ 3 & 2 \leq t < 4 \\ 1 & 4 \leq t < 8 \end{cases}

Solving Differential Equations with Discontinuous Forcing

The true power of step functions emerges when solving differential equations with discontinuous forcing—input functions that change abruptly. The Laplace transform method handles these naturally, without needing to solve separate problems on each interval.

The Method

Express the forcing function using step functions
Take the Laplace transform of both sides of the DE
Solve for $Y(s)$ algebraically
Apply the inverse Laplace transform, using the Second Shifting Theorem to handle exponential factors

Example: Spring-Mass System with Sudden Force

Problem: Solve $y'' + 4y = f(t)$ with $y(0) = 0, y'(0) = 0$ , where:

f(t) = \begin{cases} 0 & 0 \leq t < 2 \\ 3 & t \geq 2 \end{cases} = 3 \cdot u(t-2)

Physical interpretation: A spring-mass system initially at rest, subjected to a constant force of 3 that suddenly turns on at $t = 2$ .

Solution:

Step 1: Take the Laplace transform of both sides:

[s^2 Y(s) - sy(0) - y'(0)] + 4Y(s) = 3 \cdot \frac{e^{-2s}}{s}

With zero initial conditions:

(s^2 + 4)Y(s) = \frac{3e^{-2s}}{s}

Step 2: Solve for $Y(s)$ :

Y(s) = \frac{3e^{-2s}}{s(s^2 + 4)}

Step 3: Partial fractions decomposition:

\frac{3}{s(s^2+4)} = \frac{A}{s} + \frac{Bs+C}{s^2+4}

Solving: $A = 3/4$ , $B = -3/4$ , $C = 0$

\frac{3}{s(s^2+4)} = \frac{3/4}{s} - \frac{(3/4)s}{s^2+4}

Step 4: Find the inverse transform without the exponential:

\mathcal{L}^{-1}\left\{ \frac{3}{s(s^2+4)} \right\} = \frac{3}{4} - \frac{3}{4}\cos(2t)

Step 5: Apply the Second Shifting Theorem (for $e^{-2s}$ ):

y(t) = \frac{3}{4}\left[1 - \cos(2(t-2))\right] \cdot u(t-2)

This solution automatically captures the physics: the system is at rest ( $y = 0$ ) until $t = 2$ , then begins oscillating around a new equilibrium of $y = 3/4$ .

Interactive: DE Solver with Discontinuities

Explore how different discontinuous forcing functions affect the solution of a differential equation:

⚙️DE with Discontinuous Forcing

y'' + 2^2 y = 3 \cdot u(t - 2), \quad y(0) = 0, \quad y'(0) = 0

t = 0.00

ω = 2Natural frequency

A = 3Force amplitude

a = 2Force turns on at t = a

Solution:

y(t) = \frac{3}{2^2}\left[1 - \cos(2(t-2))\right] \cdot u(t-2)

The system is at rest until t = 2, then oscillates about y = 0.750

Worked Examples

Example 1: Piecewise Constant Force

Problem: Find the Laplace transform of:

f(t) = \begin{cases} 2 & 0 \leq t < 3 \\ 5 & t \geq 3 \end{cases}

Solution:

Rewrite using step functions:

f(t) = 2 + (5-2) \cdot u(t-3) = 2 + 3 \cdot u(t-3)

Take the Laplace transform:

\mathcal{L}\{f(t)\} = \frac{2}{s} + \frac{3e^{-3s}}{s}

Example 2: Ramp Starting at t = 2

Problem: Find the Laplace transform of $g(t) = (t-2) \cdot u(t-2)$ .

Solution:

This is already in the correct form $f(t-a) \cdot u(t-a)$ with $f(t) = t$ and $a = 2$ .

We know $\mathcal{L}\{t\} = 1/s^2$ , so by the Second Shifting Theorem:

\mathcal{L}\{(t-2) \cdot u(t-2)\} = \frac{e^{-2s}}{s^2}

Example 3: Quadratic Starting at t = 1

Problem: Find the Laplace transform of $h(t) = t^2 \cdot u(t-1)$ .

Solution:

Issue: The argument of $t^2$ is $t$ , not $(t-1)$ . We must rewrite.

Let $\tau = t - 1$ , so $t = \tau + 1$ :

t^2 = (\tau + 1)^2 = \tau^2 + 2\tau + 1

So:

t^2 \cdot u(t-1) = [(t-1)^2 + 2(t-1) + 1] \cdot u(t-1)

Taking the Laplace transform of each term:

$\mathcal{L}\{(t-1)^2 \cdot u(t-1)\} = \frac{2e^{-s}}{s^3}$
$\mathcal{L}\{2(t-1) \cdot u(t-1)\} = \frac{2e^{-s}}{s^2}$
$\mathcal{L}\{1 \cdot u(t-1)\} = \frac{e^{-s}}{s}$

Combining:

\mathcal{L}\{t^2 \cdot u(t-1)\} = e^{-s}\left(\frac{2}{s^3} + \frac{2}{s^2} + \frac{1}{s}\right)

Machine Learning Connections

Step functions and discontinuous signals appear throughout machine learning, often in forms you might not immediately recognize:

ReLU and the Step Function

The Rectified Linear Unit (ReLU), the most widely used activation function in deep learning, is intimately connected to the step function:

\text{ReLU}(x) = \max(0, x) = x \cdot u(x)

ReLU is literally the product of $x$ and the Heaviside step function! This connection explains:

Why ReLU creates piecewise linear decision boundaries
Why ReLU networks can approximate any continuous function
Why ReLU gradients are discontinuous (the derivative of $u(x)$ is the Dirac delta)

Learning Rate Schedules

Step learning rate schedules use the step function to reduce learning rates at specific epochs:

\eta(t) = \eta_0 \cdot \gamma^{\lfloor t/T \rfloor}

This creates a staircase pattern—exactly a sum of step functions! The Laplace transform perspective helps understand why step schedules sometimes outperform smooth schedules: they provide sharp "interventions" that can help escape local minima.

Curriculum Learning

In curriculum learning, training difficulty increases in discrete steps:

\text{difficulty}(t) = \sum_{k=1}^{K} d_k \cdot u(t - t_k)

Each step function adds harder examples at time $t_k$ . This is exactly the kind of piecewise constant function we've been studying!

Indicator Functions in Loss

Many ML losses implicitly use step/indicator functions:

Loss/Function	Step Function Connection
Hinge Loss	max(0, 1-y·f(x)) = (1-y·f(x))·u(1-y·f(x))
0-1 Loss	𝟙{y ≠ ŷ} = u(\|y-ŷ\|)
Hard Threshold	u(f(x) - θ)
Dropout Mask	Bernoulli random step functions

Python Implementation

Step Functions and Visualization

Let's implement step functions and visualize their properties:

Step Functions in Python

🐍step_functions_demo.py

Explanation(6)

Code(84)

10Symbol Setup

We define symbols with appropriate assumptions. Setting t as positive helps SymPy simplify Laplace transform results correctly.

16Heaviside Definition

The Heaviside function Heaviside(t) equals 0 for t < 0 and 1 for t ≥ 0. It models instantaneous switches or signals that turn on.

19Laplace of Step

The Laplace transform of u(t) is 1/s. This fundamental result forms the basis for handling all discontinuous functions.

29Second Shifting Theorem

This theorem states: if L{f(t)} = F(s), then L{f(t-a)·u(t-a)} = e^(-as)·F(s). The exponential factor encodes the time delay.

37NumPy Heaviside

np.heaviside(t, 1) implements the step function numerically. The second argument (1) specifies the value at t=0.

55Rectangular Pulse

A pulse of width 2 from t=1 to t=3 is expressed as u(t-1) - u(t-3). Subtracting step functions creates finite-duration signals.

78 lines without explanation

1import numpy as np
2import matplotlib.pyplot as plt
3from scipy import signal
4from sympy import *
5
6def demonstrate_step_functions():
7    """
8    Demonstrate the Heaviside step function and its properties.
9    The step function is fundamental for modeling discontinuous systems.
10    """
11    t = symbols('t', real=True, positive=True)
12    s = symbols('s')
13    a = symbols('a', positive=True)
14
15    # Define the Heaviside step function
16    print("=== Heaviside Step Function ===")
17    print("u(t) = 0 for t < 0, u(t) = 1 for t ≥ 0")
18
19    # Laplace transform of u(t)
20    u_t = Heaviside(t)
21    L_u = laplace_transform(u_t, t, s, noconds=True)
22    print(f"\nL{{u(t)}} = {L_u}")  # Should be 1/s
23
24    # Shifted step function u(t-a)
25    print("\n=== Shifted Step Function ===")
26    u_shifted = Heaviside(t - a)
27    L_u_shifted = laplace_transform(u_shifted, t, s, noconds=True)
28    print(f"L{{u(t-a)}} = {L_u_shifted}")  # Should be e^(-as)/s
29
30    # Second Shifting Theorem: L{f(t-a)·u(t-a)} = e^(-as)·F(s)
31    print("\n=== Second Shifting Theorem ===")
32    print("If L{f(t)} = F(s), then L{f(t-a)·u(t-a)} = e^(-as)·F(s)")
33
34    # Example: L{(t-2)²·u(t-2)}
35    f = (t - 2)**2 * Heaviside(t - 2)
36    L_f = laplace_transform(f, t, s, noconds=True)
37    print(f"\nL{{(t-2)²·u(t-2)}} = {L_f}")
38
39    # Numerical visualization
40    t_vals = np.linspace(-1, 5, 1000)
41    u_vals = np.heaviside(t_vals, 1)  # u(t)
42    u_shifted_vals = np.heaviside(t_vals - 2, 1)  # u(t-2)
43
44    # Create visualization
45    fig, axes = plt.subplots(2, 2, figsize=(12, 8))
46
47    # Plot u(t)
48    axes[0, 0].plot(t_vals, u_vals, 'b-', linewidth=2)
49    axes[0, 0].axhline(y=0, color='k', linewidth=0.5)
50    axes[0, 0].axvline(x=0, color='k', linewidth=0.5)
51    axes[0, 0].set_title('u(t) - Unit Step Function')
52    axes[0, 0].set_xlabel('t')
53    axes[0, 0].set_ylabel('u(t)')
54    axes[0, 0].grid(True, alpha=0.3)
55
56    # Plot u(t-2)
57    axes[0, 1].plot(t_vals, u_shifted_vals, 'r-', linewidth=2)
58    axes[0, 1].axhline(y=0, color='k', linewidth=0.5)
59    axes[0, 1].axvline(x=2, color='k', linewidth=0.5, linestyle='--')
60    axes[0, 1].set_title('u(t-2) - Shifted Step Function')
61    axes[0, 1].set_xlabel('t')
62    axes[0, 1].set_ylabel('u(t-2)')
63    axes[0, 1].grid(True, alpha=0.3)
64
65    # Rectangular pulse: u(t-1) - u(t-3)
66    pulse = np.heaviside(t_vals - 1, 1) - np.heaviside(t_vals - 3, 1)
67    axes[1, 0].plot(t_vals, pulse, 'g-', linewidth=2)
68    axes[1, 0].set_title('Rectangular Pulse: u(t-1) - u(t-3)')
69    axes[1, 0].set_xlabel('t')
70    axes[1, 0].grid(True, alpha=0.3)
71
72    # Staircase function
73    staircase = (np.heaviside(t_vals, 1) +
74                 np.heaviside(t_vals - 1, 1) +
75                 np.heaviside(t_vals - 2, 1))
76    axes[1, 1].plot(t_vals, staircase, 'm-', linewidth=2)
77    axes[1, 1].set_title('Staircase: u(t) + u(t-1) + u(t-2)')
78    axes[1, 1].set_xlabel('t')
79    axes[1, 1].grid(True, alpha=0.3)
80
81    plt.tight_layout()
82    plt.show()
83
84demonstrate_step_functions()

Solving DEs with Discontinuous Forcing

Here's how to solve differential equations with step function forcing:

Solving DEs with Step Function Forcing

🐍de_discontinuous_forcing.py

Explanation(6)

Code(101)

9The Problem

We solve y'' + 4y = f(t) where f(t) suddenly turns on at t=2. This models a spring-mass system suddenly hit by a constant force.

19Laplace Method

Taking the Laplace transform converts the ODE to an algebraic equation. The discontinuous forcing becomes e^(-2s)/s in the s-domain.

31Partial Fractions

We decompose 3/[s(s²+4)] into simpler fractions that have known inverse Laplace transforms. This is the key algebraic step.

48Analytical Solution

The solution is 0 before t=2 (the force hasn't turned on yet), then oscillates about a new equilibrium after t=2.

54SciPy Verification

We verify our analytical solution using numerical integration. The np.heaviside function handles the discontinuity in the forcing.

70Comparison

The numerical and analytical solutions match perfectly, validating both our Laplace transform method and the Second Shifting Theorem.

95 lines without explanation

1import numpy as np
2from scipy.integrate import odeint
3from scipy import signal
4import matplotlib.pyplot as plt
5from sympy import *
6
7def solve_de_with_discontinuous_forcing():
8    """
9    Solve y'' + 4y = f(t) where f(t) is a discontinuous forcing function.
10    This demonstrates how step functions handle real engineering problems.
11    """
12    t, s = symbols('t s', real=True)
13
14    # Define the forcing function:
15    # f(t) = 0 for t < 2, f(t) = 3 for t >= 2
16    # Written as: f(t) = 3·u(t-2)
17    print("=== Discontinuous Forcing Problem ===")
18    print("y'' + 4y = f(t), y(0) = 0, y'(0) = 0")
19    print("where f(t) = 3·u(t-2)")
20
21    # Method: Laplace Transform
22    # L{y''} + 4L{y} = L{3·u(t-2)}
23    # s²Y(s) - sy(0) - y'(0) + 4Y(s) = 3·e^(-2s)/s
24    # (s² + 4)Y(s) = 3·e^(-2s)/s
25    # Y(s) = 3·e^(-2s) / [s(s² + 4)]
26
27    print("\n=== Laplace Transform Method ===")
28    print("Taking L of both sides:")
29    print("(s² + 4)Y(s) = 3e^(-2s)/s")
30    print("Y(s) = 3e^(-2s) / [s(s² + 4)]")
31
32    # Partial fractions for 3/[s(s²+4)]
33    # 3/[s(s²+4)] = A/s + (Bs+C)/(s²+4)
34    # 3 = A(s²+4) + (Bs+C)s
35    # A = 3/4, B = -3/4, C = 0
36    # So: 3/[s(s²+4)] = (3/4)/s - (3/4)s/(s²+4)
37
38    print("\nPartial fractions:")
39    print("3/[s(s²+4)] = (3/4)/s - (3/4)s/(s²+4)")
40
41    # Inverse Laplace:
42    # L⁻¹{(3/4)/s} = 3/4
43    # L⁻¹{(3/4)s/(s²+4)} = (3/4)cos(2t)
44    # So L⁻¹{3/[s(s²+4)]} = (3/4) - (3/4)cos(2t) = (3/4)(1 - cos(2t))
45
46    # By Second Shifting Theorem:
47    # y(t) = (3/4)(1 - cos(2(t-2)))·u(t-2)
48
49    print("\nSolution:")
50    print("y(t) = (3/4)(1 - cos(2(t-2)))·u(t-2)")
51    print("     = 0                           for t < 2")
52    print("     = (3/4)(1 - cos(2(t-2)))      for t ≥ 2")
53
54    # Numerical verification with scipy
55    def forcing(t):
56        return 3 * np.heaviside(t - 2, 1)
57
58    def ode_system(y, t):
59        # y[0] = y, y[1] = y'
60        # y' = y[1]
61        # y'' = f(t) - 4*y[0]
62        return [y[1], forcing(t) - 4*y[0]]
63
64    t_span = np.linspace(0, 10, 1000)
65    y0 = [0, 0]  # Initial conditions
66    solution = odeint(ode_system, y0, t_span)
67
68    # Analytical solution
69    def analytical_solution(t):
70        return np.where(t < 2, 0,
71                       (3/4) * (1 - np.cos(2*(t - 2))))
72
73    # Plot comparison
74    fig, axes = plt.subplots(2, 1, figsize=(10, 8))
75
76    # Plot forcing function
77    axes[0].plot(t_span, forcing(t_span), 'b-', linewidth=2)
78    axes[0].set_title('Forcing Function: f(t) = 3·u(t-2)')
79    axes[0].set_xlabel('t')
80    axes[0].set_ylabel('f(t)')
81    axes[0].grid(True, alpha=0.3)
82    axes[0].axvline(x=2, color='r', linestyle='--', label='t = 2')
83    axes[0].legend()
84
85    # Plot solution
86    axes[1].plot(t_span, solution[:, 0], 'b-',
87                 linewidth=2, label='Numerical (scipy)')
88    axes[1].plot(t_span, analytical_solution(t_span), 'r--',
89                 linewidth=2, label='Analytical')
90    axes[1].set_title('Solution: y(t)')
91    axes[1].set_xlabel('t')
92    axes[1].set_ylabel('y(t)')
93    axes[1].grid(True, alpha=0.3)
94    axes[1].legend()
95
96    plt.tight_layout()
97    plt.show()
98
99    return solution
100
101solve_de_with_discontinuous_forcing()

Step Functions in Machine Learning

Step functions appear throughout ML in various disguises:

Step Functions in Machine Learning

🐍step_functions_ml.py

Explanation(5)

Code(106)

15ReLU = x · u(x)

The ReLU activation function is mathematically equivalent to x times the Heaviside step function! This connection explains its piecewise linear nature.

32Step Learning Rate

Step decay schedules reduce the learning rate by a factor at discrete epochs. This creates staircase-like LR curves using step function logic.

49Hard Sigmoid

Hard sigmoid uses piecewise linear segments (defined by step conditions) to approximate the smooth sigmoid. This is faster to compute in hardware.

64Curriculum Learning

In curriculum learning, training difficulty often increases in discrete steps. Each step adds harder examples, modeled as sum of shifted step functions.

87Indicator Functions

Many ML losses implicitly use step/indicator functions: hinge loss uses max(0, ·), 0-1 loss is a step function, and margin losses have discontinuities.

101 lines without explanation

1import numpy as np
2import matplotlib.pyplot as plt
3
4def step_functions_in_ml():
5    """
6    Step functions appear throughout machine learning in various forms:
7    - Activation functions (ReLU, Hard Sigmoid)
8    - Learning rate schedules
9    - Curriculum learning
10    - Indicator functions in loss functions
11    """
12
13    # 1. ReLU as step function composition
14    print("=== ReLU and Step Functions ===")
15    print("ReLU(x) = max(0, x) = x · u(x)")
16    print("ReLU is the product of x and the step function!")
17
18    x = np.linspace(-5, 5, 1000)
19    relu = np.maximum(0, x)
20    step = np.heaviside(x, 0.5)
21    x_times_step = x * step
22
23    fig, axes = plt.subplots(2, 2, figsize=(12, 10))
24
25    # ReLU = x · u(x)
26    axes[0, 0].plot(x, relu, 'b-', linewidth=2, label='ReLU(x)')
27    axes[0, 0].plot(x, x_times_step, 'r--', linewidth=2, label='x·u(x)')
28    axes[0, 0].set_title('ReLU(x) = x · u(x)')
29    axes[0, 0].legend()
30    axes[0, 0].grid(True, alpha=0.3)
31    axes[0, 0].set_xlabel('x')
32
33    # 2. Learning Rate Schedules with Steps
34    print("\n=== Step Learning Rate Schedules ===")
35    epochs = np.arange(0, 100)
36
37    # Step decay: reduce LR by factor every 30 epochs
38    lr_initial = 0.1
39    lr_step = lr_initial * (0.1 ** np.floor(epochs / 30))
40
41    # Warmup + step decay
42    warmup_epochs = 10
43    warmup = np.minimum(epochs / warmup_epochs, 1)
44    lr_warmup_step = warmup * lr_step
45
46    axes[0, 1].plot(epochs, lr_step, 'b-', linewidth=2, label='Step Decay')
47    axes[0, 1].plot(epochs, lr_warmup_step, 'r-', linewidth=2, label='Warmup + Step')
48    axes[0, 1].set_title('Step Learning Rate Schedules')
49    axes[0, 1].set_xlabel('Epoch')
50    axes[0, 1].set_ylabel('Learning Rate')
51    axes[0, 1].legend()
52    axes[0, 1].grid(True, alpha=0.3)
53    axes[0, 1].set_yscale('log')
54
55    # 3. Hard Sigmoid (approximation of sigmoid with steps)
56    print("\n=== Hard Sigmoid ===")
57    print("HardSigmoid(x) = clip((x + 3)/6, 0, 1)")
58    print("This is a piecewise linear approximation using step logic")
59
60    sigmoid = 1 / (1 + np.exp(-x))
61    hard_sigmoid = np.clip((x + 3) / 6, 0, 1)
62
63    axes[1, 0].plot(x, sigmoid, 'b-', linewidth=2, label='Sigmoid')
64    axes[1, 0].plot(x, hard_sigmoid, 'r-', linewidth=2, label='Hard Sigmoid')
65    axes[1, 0].set_title('Sigmoid vs Hard Sigmoid')
66    axes[1, 0].legend()
67    axes[1, 0].grid(True, alpha=0.3)
68    axes[1, 0].set_xlabel('x')
69
70    # 4. Curriculum Learning (task difficulty)
71    print("\n=== Curriculum Learning ===")
72    print("Task difficulty increases in discrete steps")
73
74    training_step = np.arange(0, 10000)
75
76    # Step-based difficulty
77    difficulty_steps = (
78        0.3 * np.heaviside(training_step, 1) +
79        0.3 * np.heaviside(training_step - 3000, 1) +
80        0.4 * np.heaviside(training_step - 6000, 1)
81    )
82
83    # Smooth version for comparison
84    difficulty_smooth = 0.3 + 0.7 * (1 - np.exp(-training_step / 3000))
85
86    axes[1, 1].plot(training_step, difficulty_steps, 'b-',
87                    linewidth=2, label='Step Curriculum')
88    axes[1, 1].plot(training_step, difficulty_smooth, 'r--',
89                    linewidth=2, label='Smooth Curriculum')
90    axes[1, 1].set_title('Curriculum Learning: Task Difficulty')
91    axes[1, 1].set_xlabel('Training Step')
92    axes[1, 1].set_ylabel('Difficulty')
93    axes[1, 1].legend()
94    axes[1, 1].grid(True, alpha=0.3)
95
96    plt.tight_layout()
97    plt.show()
98
99    # 5. Indicator Functions in Loss
100    print("\n=== Indicator Functions in Loss ===")
101    print("Many losses use implicit step/indicator functions:")
102    print("- Hinge Loss: max(0, 1-y·f(x)) uses step logic")
103    print("- 0-1 Loss: I{y ≠ ŷ} is a step function")
104    print("- Margin-based losses: discontinuities at margins")
105
106step_functions_in_ml()

Common Mistakes to Avoid

Mistake 1: Forgetting to Rewrite Before Shifting

Wrong: Applying the Second Shifting Theorem directly to $t^2 \cdot u(t-3)$

Correct: First rewrite as $[(t-3)+3]^2 \cdot u(t-3)$ , expand, then apply the theorem to each term.

The argument of the function must match the argument of the step function!

Mistake 2: Getting the Shift Direction Wrong

Wrong: Thinking $u(t-a)$ shifts the function left

Correct: $u(t-a)$ turns on at $t = a$ , which is a shift right by $a$ units.

Remember: replacing $t$ with $t-a$ always shifts right (positive $a$ ).

Mistake 3: Confusing the Two Shifting Theorems

First Shifting Theorem: $\mathcal{L}\{e^{at}f(t)\} = F(s-a)$ (s-domain shift)

Second Shifting Theorem: $\mathcal{L}\{f(t-a)u(t-a)\} = e^{-as}F(s)$ (t-domain shift)

The first involves multiplication by $e^{at}$ in time; the second involves time delay with step functions.

Mistake 4: Incorrect Partial Fractions with Exponentials

Wrong: Including $e^{-as}$ in partial fractions

Correct: Factor out $e^{-as}$ , do partial fractions on the rational part, then multiply back.

The exponential factor passes through the inverse Laplace transform via the Second Shifting Theorem—it doesn't participate in partial fractions.

Test Your Understanding

📝Test Your Understanding

Question 1 of 5

What is the Laplace transform of the shifted step function u(t-3)?

Summary

Step functions are the mathematical key to handling discontinuous signals and forcing functions in differential equations. Combined with Laplace transforms, they provide a powerful, systematic approach to solving problems that would be extremely difficult by other methods.

Key Formulas

Formula	Name	Use
u(t-a)	Shifted step function	Turns on at t = a
𝓛{u(t)} = 1/s	Transform of step	Basic result
𝓛{u(t-a)} = e^(-as)/s	Transform of shifted step	Time delay
𝓛{f(t-a)·u(t-a)} = e^(-as)F(s)	Second Shifting Theorem	Key theorem
𝓛⁻¹{e^(-as)F(s)} = f(t-a)·u(t-a)	Inverse Second Shifting	Inverting delays

Key Takeaways

Step functions model switches: The Heaviside function $u(t-a)$ represents a signal that turns on at time $a$ .
Exponentials encode delays: The factor $e^{-as}$ in the s-domain always corresponds to a time shift of $a$ in the time domain.
Piecewise functions become sums: Any piecewise function can be written as a sum of terms involving step functions.
Match arguments for shifting: To use the Second Shifting Theorem, the function argument must be $(t-a)$ to match the step function $u(t-a)$ .
Factor out exponentials: When doing partial fractions, factor out $e^{-as}$ first, work with the rational part, then apply the shifting theorem.
ML is full of steps: ReLU, learning rate schedules, curriculum learning, and many loss functions all involve step function logic.

The Core Insight:

"The step function is the mathematical switch—it turns signals on and off, enabling us to model the discontinuous reality of engineering and physics with elegant precision."

Coming Next: In Impulse Functions and the Delta Function, we'll encounter an even more singular object—the Dirac delta function, which represents an instantaneous impulse and is the derivative of the step function.