Boo-AI — Master Artificial Intelligence by Building from Scratch

Learning Objectives

By the end of this section, you will be able to:

Understand the accumulation function $F(x) = \int_a^x f(t) \, dt$ and its geometric meaning
State and explain the Fundamental Theorem of Calculus Part 1
Prove intuitively why $F'(x) = f(x)$ using the thin strip argument
Apply FTC Part 1 to find derivatives of integral functions
Extend the theorem using the chain rule for variable limits
Connect this theorem to probability (CDF/PDF relationship) and machine learning

The Big Picture: Bridging Two Worlds

"The Fundamental Theorem of Calculus is the spine of calculus — it connects the two halves (differentiation and integration) into one coherent body."

Throughout history, mathematicians developed two seemingly unrelated operations:

Differentiation: Finding instantaneous rates of change — how fast things are changing at any moment
Integration: Computing areas and accumulations — the total effect of continuous change

For centuries, these were studied as separate subjects. Then came the remarkable discovery: they are inverse operations. The Fundamental Theorem of Calculus (FTC) reveals this profound connection, and it comes in two parts:

FTC Part 1 (This Section)

Integration then Differentiation

If you integrate a function and then differentiate the result, you get the original function back.

\frac{d}{dx} \int_a^x f(t) \, dt = f(x)

FTC Part 2 (Next Section)

Differentiation then Integration

If you find an antiderivative and evaluate at the endpoints, you get the definite integral.

\int_a^b f(x) \, dx = F(b) - F(a)

Why This Matters

The FTC is not just a theoretical result — it's a computational breakthrough. Before the FTC, calculating areas required tedious limit computations with Riemann sums. After the FTC, we can find areas simply by finding antiderivatives — a far easier task in most cases.

Historical Context: A Revolutionary Insight

The Fundamental Theorem emerged from the work of two mathematical giants in the 17th century:

Isaac Newton (1643–1727)

Newton developed his "method of fluxions" around 1666 while escaping the plague in the English countryside. He viewed derivatives as velocities (rates of change) and integrals as areas under velocity curves (accumulated distance). His key insight was that finding the area under a velocity curve gives you the total distance traveled — connecting integration to antidifferentiation.

Gottfried Wilhelm Leibniz (1646–1716)

Working independently in Germany, Leibniz developed the notation we use today: $\int$ for integration and $\frac{dy}{dx}$ for derivatives. He understood the integral as a "sum" of infinitesimal rectangles and recognized that differentiation "undoes" this summation.

The Controversy

Newton and Leibniz independently discovered calculus, leading to one of history's bitterest priority disputes. Today, we recognize both as co-discoverers, and we use Leibniz's notation while appreciating Newton's physical intuition.

The Physical Insight

Consider a car traveling along a road. Let $v(t)$ be its velocity at time $t$ .

The position at time $t$ is $s(t) = \int_0^t v(\tau) \, d\tau$ — the accumulated distance
The velocity is the rate of change of position: $v(t) = s'(t)$

Combining these: $\frac{d}{dt} \int_0^t v(\tau) \, d\tau = v(t)$ . This is exactly FTC Part 1! The rate at which distance accumulates is the instantaneous velocity.

The Accumulation Function

Before stating the FTC, we need to understand a special kind of function called the accumulation function (also known as the area function).

Definition: The Accumulation Function

F(x) = \int_a^x f(t) \, dt

What each symbol means:

$a$ : A fixed starting point (lower limit)
$x$ : A variable endpoint (upper limit) — this is the input to $F$
$t$ : A "dummy variable" of integration (could be any letter)
$f(t)$ : The function being integrated (assumed continuous)
$F(x)$ : The accumulated area under $f$ from $a$ to $x$

What Does F(x) Measure?

For each value of $x$ , $F(x)$ gives the signed area under the curve $y = f(t)$ from $t = a$ to $t = x$ .

When $x = a$ : $F(a) = \int_a^a f(t) \, dt = 0$ (no area yet)
As $x$ increases, more area is included, so $F(x)$ grows (if $f > 0$ )
If $f(t) < 0$ for some $t$ , that region contributes negative area

Key Insight

The accumulation function $F(x)$ transforms a rate function $f$ into a total function. If $f(t)$ represents a rate (like velocity), then $F(x)$ represents the cumulative total (like distance traveled).

Interactive: The Accumulation Function

Explore how the accumulation function works. Drag the upper limit $x$ and watch how the shaded area (and hence $F(x)$ ) changes:

The Accumulation Function: F(x) = ∫_a^x f(t) dt

Function:

Lower bound a:0.00

Upper bound x:2.00

Original function f(t) with shaded area = F(x)

Accumulation function F(x) = ∫_a^x f(t) dt

Accumulation Value

F(2.00) = 2.6667

Area under f(t) from 0.0 to 2.00

Derivative (FTC Part 1)

F'(2.00) = 4.0000

Rate of change of area = height of f at x

The Fundamental Insight

As you drag x to the right, the shaded area grows. The rate at which this area grows is exactly the height of the function f at that point. This is why F'(x) = f(x).

FTC Part 1: The Statement

The Fundamental Theorem of Calculus, Part 1

If $f$ is continuous on $[a, b]$ , then the function

F(x) = \int_a^x f(t) \, dt

is continuous on $[a, b]$ , differentiable on $(a, b)$ , and

F'(x) = f(x)

Equivalently: $\frac{d}{dx} \int_a^x f(t) \, dt = f(x)$

What This Theorem Says

In plain language: differentiation undoes integration. If you start with a function $f$ , integrate it to get the area function $F$ , and then differentiate $F$ , you get back the original $f$ .

More intuitively: the rate at which area accumulates equals the height of the curve. As you move the upper limit $x$ to the right, the area grows at a rate equal to the current height $f(x)$ .

Intuitive Proof: The Thin Strip Argument

Let's understand why FTC Part 1 is true through geometric reasoning.

Consider the accumulation function $F(x) = \int_a^x f(t) \, dt$ . We want to find $F'(x)$ , the instantaneous rate of change of the area.

Step 1: The Difference Quotient

By definition of the derivative:

F'(x) = \lim_{\Delta x \to 0} \frac{F(x + \Delta x) - F(x)}{\Delta x}

Step 2: Interpret F(x + Δx) - F(x)

Using the additivity property of integrals:

F(x + \Delta x) - F(x) = \int_a^{x + \Delta x} f(t) \, dt - \int_a^x f(t) \, dt = \int_x^{x + \Delta x} f(t) \, dt

This is the area of a thin strip of width $\Delta x$ under the curve.

Step 3: Approximate the Strip

When $\Delta x$ is very small, the thin strip is approximately a rectangle with:

Width: $\Delta x$
Height: approximately $f(x)$ (the height at the left edge)

So the area of the strip is approximately:

\int_x^{x + \Delta x} f(t) \, dt \approx f(x) \cdot \Delta x

Step 4: Take the Limit

Substituting into the difference quotient:

F'(x) = \lim_{\Delta x \to 0} \frac{f(x) \cdot \Delta x}{\Delta x} = \lim_{\Delta x \to 0} f(x) = f(x)

The approximation becomes exact as $\Delta x \to 0$ because $f$ is continuous.

Interactive: The Thin Strip Argument

Explore the thin strip argument visually. Shrink $\Delta x$ and watch how the strip area divided by $\Delta x$ approaches $f(x)$ :

Why F'(x) = f(x): The Thin Strip Argument

Position x:2.00

Strip width \u0394x:0.300

Height at x

f(2.00) = 1.700

Strip Area (exact)

\u0394F = 0.5667

Rectangle Approx

f(x)\u00B7\u0394x = 0.5100

Rate \u0394F/\u0394x

1.8890

The FTC Part 1 Derivation

The green strip has area \u0394F = F(x + \u0394x) - F(x). When \u0394x is small:

\u0394F \u2248 f(x) \u00B7 \u0394x

\u21D2 \u0394F/\u0394x \u2248 f(x)

⇒ lim_Δx→0 ΔF/Δx = F'(x) = f(x)

Current approximation error: 0.056705

Convergence as \u0394x \u2192 0

Exact rate \u0394F/\u0394x

1.88902

True derivative f(x)

1.70000

Difference

0.18902

As \u0394x gets smaller, \u0394F/\u0394x converges to exactly f(x) - this is the FTC!

Formal Proof

For completeness, here is the rigorous proof using the definition of the derivative and properties of continuous functions.

Proof of FTC Part 1

Given: $f$ is continuous on $[a, b]$ and $F(x) = \int_a^x f(t) \, dt$ .

To prove: $F'(x) = f(x)$ for all $x \in (a, b)$ .

Proof:

For $\Delta x \neq 0$ small enough that $x + \Delta x \in [a, b]$ :

\frac{F(x + \Delta x) - F(x)}{\Delta x} = \frac{1}{\Delta x} \int_x^{x + \Delta x} f(t) \, dt

By the Mean Value Theorem for Integrals, since $f$ is continuous, there exists $c$ between $x$ and $x + \Delta x$ such that:

\int_x^{x + \Delta x} f(t) \, dt = f(c) \cdot \Delta x

Therefore:

\frac{F(x + \Delta x) - F(x)}{\Delta x} = f(c)

As $\Delta x \to 0$ , we have $c \to x$ (since $c$ is squeezed between $x$ and $x + \Delta x$ ).

Since $f$ is continuous, $f(c) \to f(x)$ as $c \to x$ .

Therefore:

F'(x) = \lim_{\Delta x \to 0} f(c) = f(x) \quad \blacksquare

Worked Examples

Example 1: Basic Application

Problem: Let $F(x) = \int_0^x t^3 \, dt$ . Find $F'(x)$ .

Solution: By FTC Part 1, since $f(t) = t^3$ is continuous:

F'(x) = x^3

Notice: We don't need to evaluate the integral first! FTC Part 1 tells us the derivative directly.

Example 2: Trigonometric Integrand

Problem: Let $G(x) = \int_1^x \cos(t^2) \, dt$ . Find $G'(x)$ .

Solution: Even though $\cos(t^2)$ has no elementary antiderivative, FTC Part 1 gives:

G'(x) = \cos(x^2)

This demonstrates the power of FTC Part 1 — we can differentiate integral functions even when we can't integrate explicitly.

Example 3: Finding Specific Values

Problem: Let $H(x) = \int_2^x e^{-t^2} \, dt$ . Find $H'(3)$ and $H(2)$ .

Solution:

By FTC Part 1: $H'(x) = e^{-x^2}$

So $H'(3) = e^{-9} \approx 0.000123$

For $H(2)$ : $H(2) = \int_2^2 e^{-t^2} \, dt = 0$ (zero-width integral)

Chain Rule Extension

What if the upper limit is not simply $x$ , but a function of $x$ ? We combine FTC Part 1 with the chain rule.

FTC Part 1 with Chain Rule

\frac{d}{dx} \int_a^{g(x)} f(t) \, dt = f(g(x)) \cdot g'(x)

Example: Variable Upper Limit

Problem: Find $\frac{d}{dx} \int_0^{x^2} \sin(t) \, dt$

Solution:

Let $g(x) = x^2$ , so $g'(x) = 2x$ .

By the chain rule version of FTC Part 1:

\frac{d}{dx} \int_0^{x^2} \sin(t) \, dt = \sin(x^2) \cdot 2x = 2x\sin(x^2)

Example: Variable Lower and Upper Limits

Problem: Find $\frac{d}{dx} \int_x^{x^3} e^{t^2} \, dt$

Solution: Split using additivity:

\int_x^{x^3} e^{t^2} \, dt = \int_0^{x^3} e^{t^2} \, dt - \int_0^x e^{t^2} \, dt

Differentiating each term:

\frac{d}{dx} = e^{(x^3)^2} \cdot 3x^2 - e^{x^2} \cdot 1 = 3x^2 e^{x^6} - e^{x^2}

Real-World Applications

Physics: Position from Velocity

If $v(t)$ is velocity, then position at time $t$ is:

s(t) = s_0 + \int_0^t v(\tau) \, d\tau

FTC Part 1 confirms: $s'(t) = v(t)$ — the derivative of position is velocity.

Economics: Total Cost from Marginal Cost

If $MC(q)$ is marginal cost (cost of producing one more unit), then total cost of producing $q$ units beyond a base level is:

TC(q) = TC_0 + \int_0^q MC(x) \, dx

FTC Part 1 confirms: $TC'(q) = MC(q)$ — marginal cost is the derivative of total cost.

Biology: Accumulated Drug Concentration

If $r(t)$ is the rate of drug absorption, the total drug in the body at time $t$ is:

D(t) = \int_0^t r(\tau) \, d\tau

FTC Part 1 tells us $D'(t) = r(t)$ — the rate of change of total drug equals the absorption rate.

Machine Learning Connection

The Fundamental Theorem of Calculus Part 1 appears throughout machine learning and statistics in several important ways.

The CDF-PDF Relationship

In probability theory, if $X$ is a continuous random variable with probability density function (PDF) $f(x)$ , then its cumulative distribution function (CDF) is:

F(x) = P(X \leq x) = \int_{-\infty}^x f(t) \, dt

By FTC Part 1: $F'(x) = f(x)$ — the derivative of the CDF is the PDF. This fundamental relationship is used constantly in:

Density estimation: Learning probability distributions from data
Normalizing flows: Transforming simple distributions into complex ones
Quantile functions: Inverse sampling for Monte Carlo methods

Gradients of Expected Loss

In ML, loss functions are often expectations (integrals over distributions):

L(\theta) = \mathbb{E}_{x \sim p}[\ell(\theta, x)] = \int \ell(\theta, x) \, p(x) \, dx

To optimize with gradient descent, we need $\nabla_\theta L$ . Under certain conditions (Leibniz integral rule, related to FTC), we can differentiate under the integral:

\nabla_\theta L = \int \nabla_\theta \ell(\theta, x) \, p(x) \, dx

Score Functions and Fisher Information

In maximum likelihood estimation, the score function is:

s(\theta) = \nabla_\theta \log p(x | \theta)

The expected score is zero: $\mathbb{E}[s(\theta)] = \int s(\theta) \, p(x|\theta) \, dx = 0$ . This follows from FTC Part 1 applied to the condition that probability integrates to 1.

Reparameterization Trick

The reparameterization trick in Variational Autoencoders (VAEs) relies on being able to move derivatives inside expectations. This is essentially an application of the ideas behind FTC Part 1 to stochastic gradients.

Python Implementation

Verifying FTC Part 1 Numerically

Let's verify that $F'(x) = f(x)$ using numerical methods:

Verifying FTC Part 1

🐍ftc_part1_verification.py

Explanation(4)

Code(39)

8Integrand Definition

We define f(t) = sin(t) + t², the function we will integrate. FTC Part 1 says the derivative of its integral will give us back this function.

12Accumulation Function F(x)

F(x) = ∫[0 to x] f(t) dt is computed using numerical integration. This is the 'area so far' function that accumulates as x increases.

23Numerical Derivative

We compute F'(x) using the central difference method: F'(x) ≈ [F(x+h) - F(x-h)] / 2h. This should equal f(x) by FTC Part 1.

26Verification

The error between F'(x) and f(x) is extremely small (around 10⁻¹⁰), confirming that F'(x) = f(x) as the FTC states.

35 lines without explanation

1import numpy as np
2from scipy import integrate, misc
3
4def verify_ftc_part1():
5    """
6    Verify the Fundamental Theorem of Calculus Part 1:
7    If F(x) = ∫[a to x] f(t) dt, then F'(x) = f(x)
8    """
9    # Define our integrand f(t)
10    def f(t):
11        return np.sin(t) + t**2
12
13    # Define the accumulation function F(x) = ∫[0 to x] f(t) dt
14    def F(x):
15        result, _ = integrate.quad(f, 0, x)
16        return result
17
18    # Test at several points
19    test_points = [0.5, 1.0, 1.5, 2.0, 2.5]
20
21    print("Verifying FTC Part 1: F'(x) = f(x)")
22    print("=" * 50)
23    print(f"{'x':>6} | {'F\'(x) (numerical)':>20} | {'f(x)':>12} | {'Error':>12}")
24    print("-" * 50)
25
26    for x in test_points:
27        # Compute F'(x) numerically using central difference
28        F_prime_numerical = misc.derivative(F, x, dx=1e-6)
29
30        # Compare with f(x)
31        f_at_x = f(x)
32        error = abs(F_prime_numerical - f_at_x)
33
34        print(f"{x:>6.1f} | {F_prime_numerical:>20.10f} | {f_at_x:>12.10f} | {error:>12.2e}")
35
36    print("=" * 50)
37    print("The numerical derivative F'(x) matches f(x) closely!")
38
39verify_ftc_part1()

FTC Part 1 in Machine Learning

Here's how FTC Part 1 appears in probability and ML contexts:

FTC in Probability and ML

🐍ftc_ml_applications.py

Explanation(5)

Code(79)

7CDF-PDF Relationship

The CDF F(x) = P(X ≤ x) is the integral of the PDF from -∞ to x. By FTC Part 1, differentiating the CDF gives back the PDF.

17CDF as Accumulation Function

The CDF is exactly F(x) = ∫_{-∞}^x f(t) dt where f is the PDF. This is the accumulation function from FTC Part 1.

30Verifying the Relationship

We numerically differentiate the CDF and compare it to the PDF. They match closely, confirming FTC Part 1 in the probability context.

45Gradients of Expected Loss

In ML, loss functions are often expectations (integrals). FTC Part 1 justifies taking gradients through these expectations - essential for gradient descent.

58Leibniz Integral Rule

The ability to differentiate under the integral sign (Leibniz rule) is closely related to FTC Part 1 and is fundamental for computing gradients in ML.

74 lines without explanation

1import numpy as np
2from scipy import integrate
3
4def cdf_pdf_relationship():
5    """
6    Demonstrate the FTC Part 1 in probability:
7    The derivative of the CDF is the PDF.
8
9    F(x) = P(X ≤ x) = ∫[-∞ to x] f(t) dt
10    F'(x) = f(x) (the PDF)
11    """
12    def gaussian_pdf(t, mu=0, sigma=1):
13        """Standard normal PDF"""
14        return (1 / (np.sqrt(2 * np.pi) * sigma)) * \
15               np.exp(-((t - mu)**2) / (2 * sigma**2))
16
17    def gaussian_cdf(x, mu=0, sigma=1):
18        """CDF computed as integral of PDF"""
19        result, _ = integrate.quad(
20            lambda t: gaussian_pdf(t, mu, sigma),
21            -10, x  # -10 approximates -∞
22        )
23        return result
24
25    # Verify that d/dx CDF(x) = PDF(x)
26    print("FTC Part 1 in Probability: d/dx CDF(x) = PDF(x)")
27    print("=" * 55)
28
29    test_points = [-2, -1, 0, 1, 2]
30    h = 1e-6
31
32    for x in test_points:
33        # Numerical derivative of CDF
34        cdf_derivative = (gaussian_cdf(x + h) - gaussian_cdf(x - h)) / (2 * h)
35
36        # PDF at x
37        pdf_at_x = gaussian_pdf(x)
38
39        print(f"x = {x:>2}: CDF'(x) = {cdf_derivative:.8f}, "
40              f"PDF(x) = {pdf_at_x:.8f}")
41
42    print("\nThis is why we can 'differentiate the CDF to get the PDF'!")
43
44def softmax_gradient_integral():
45    """
46    In ML, loss functions often involve integrals over probability densities.
47    FTC Part 1 helps compute gradients efficiently.
48    """
49    print("\n" + "=" * 55)
50    print("FTC in ML: Gradient of Expected Loss")
51    print("=" * 55)
52
53    # Expected loss: E[L(θ)] = ∫ L(θ, x) p(x) dx
54    # By FTC, we can differentiate under the integral
55
56    def expected_loss_gradient():
57        # For a simple example: L(θ) = E[(X - θ)²]
58        # ∂/∂θ E[(X - θ)²] = E[-2(X - θ)] = -2(E[X] - θ)
59
60        # With Gaussian X ~ N(μ, σ²)
61        mu, sigma = 1.0, 2.0
62        theta = 0.5
63
64        # Gradient: -2(μ - θ)
65        analytical_grad = -2 * (mu - theta)
66
67        # Numerical verification via Monte Carlo
68        n_samples = 100000
69        samples = np.random.normal(mu, sigma, n_samples)
70        numerical_grad = np.mean(-2 * (samples - theta))
71
72        print(f"Analytical gradient: {analytical_grad:.6f}")
73        print(f"Monte Carlo estimate: {numerical_grad:.6f}")
74        print("The FTC lets us interchange differentiation and integration!")
75
76    expected_loss_gradient()
77
78cdf_pdf_relationship()
79softmax_gradient_integral()

Common Mistakes to Avoid

Mistake 1: Forgetting the Chain Rule

Wrong: $\frac{d}{dx} \int_0^{x^2} t^3 \, dt = (x^2)^3 = x^6$

Correct: $\frac{d}{dx} \int_0^{x^2} t^3 \, dt = (x^2)^3 \cdot 2x = 2x^7$

When the upper limit is a function of $x$ , multiply by its derivative.

Mistake 2: Confusing the Variable of Integration

Wrong: In $F(x) = \int_0^x t^2 \, dt$ , thinking that $F'(x) = 2t$

Correct: $F'(x) = x^2$

Replace the dummy variable $t$ with $x$ after applying FTC Part 1.

Mistake 3: Ignoring the Continuity Requirement

FTC Part 1 requires $f$ to be continuous. If $f$ has discontinuities, the theorem may not apply directly.

For piecewise continuous functions, apply the theorem on each continuous piece separately.

Mistake 4: Applying FTC Part 1 to Part 2 Problems

FTC Part 1: Differentiating an integral with variable upper limit

FTC Part 2: Evaluating a definite integral using antiderivatives

These are different applications! Part 1 tells us about derivatives; Part 2 tells us how to compute integrals.

Test Your Understanding

Test Your Understanding: FTC Part 1

1. If F(x) = ∫₀ˣ t² dt, what is F'(x) according to FTC Part 1?

2. What does F(x) = ∫ₐˣ f(t) dt represent geometrically?

3. If G(x) = ∫₀ˣ sin(t) dt, what is G'(π/2)?

4. Why must f be continuous for FTC Part 1 to apply?

5. If H(x) = ∫₃ˣ eᵗ dt, what is H(3)?

6. What is the intuitive reason F'(x) = f(x)?

7. If F(x) = ∫₀ˣ (3t² + 2t) dt, find F'(2).

8. In machine learning, the CDF F(x) = P(X ≤ x) = ∫_{-∞}ˣ f(t) dt has derivative F'(x) equal to what?

Summary

The Fundamental Theorem of Calculus Part 1 establishes the profound connection between differentiation and integration — they are inverse operations.

Key Results

Concept	Formula	Meaning
Accumulation Function	F(x) = ∫ₐˣ f(t) dt	Area under f from a to x
FTC Part 1	F'(x) = f(x)	Derivative of area = height
Chain Rule Version	d/dx ∫ₐ^{g(x)} f(t) dt = f(g(x))·g'(x)	For variable upper limits
CDF-PDF Relationship	F'(x) = f(x) where F is CDF	Derivative of CDF is PDF

Key Takeaways

Differentiation undoes integration: If you integrate a continuous function then differentiate, you recover the original function.
Geometric interpretation: The rate at which area accumulates equals the height of the curve at that point.
Practical power: We can differentiate integral functions without evaluating the integral explicitly.
Chain rule extension: When the limit is a function, multiply by its derivative.
ML connections: FTC Part 1 underlies the CDF-PDF relationship and enables differentiation through expectations.

The Core Insight:

"The derivative of the accumulated area under a curve equals the height of the curve — integration and differentiation are inverse processes."

Coming Next: In FTC Part 2, we'll discover how this relationship gives us a powerful method for computing definite integrals: just find an antiderivative and evaluate at the endpoints!