Boo-AI — Master Artificial Intelligence by Building from Scratch

Learning Objectives

By the end of this section, you will be able to:

Understand that the derivative is not just a number at one point, but a new function that outputs slopes at every point
Define the derivative function $f'(x)$ formally using limit notation
Use different notations for the derivative (Leibniz, Lagrange, Newton)
Compute derivative functions from the limit definition
Analyze where a function is increasing or decreasing using the sign of its derivative
Sketch the graph of a derivative given the graph of a function
Connect the derivative function to gradient computations in machine learning

The Big Picture: A Function That Outputs Slopes

"The derivative transforms one function into another, mapping each input to the slope of the original at that point."

In the previous section, we learned how to compute the derivative $f'(a)$ at a single point $x = a$ . This gave us the instantaneous rate of change — the slope of the tangent line — at that specific location.

But here's the crucial insight: we can compute this slope at every point where the function is differentiable. Instead of asking "What is the slope at $x = 2$ ?", we can ask "What is the slope at any point x?"

The answer is a new function — the derivative function $f'(x)$ .

The Key Conceptual Shift

Before: The derivative at a point is a number (the slope at that point)

Now: The derivative function is a function that outputs slopes for any input

$f: \text{inputs} \to \text{outputs}$

$f': \text{inputs} \to \text{slopes of } f \text{ at those inputs}$

From Number to Function: The Derivative Transformation

Consider the function $f(x) = x^2$ . In the previous section, we computed:

$f'(1) = 2$ (slope at $x = 1$ )
$f'(2) = 4$ (slope at $x = 2$ )
$f'(3) = 6$ (slope at $x = 3$ )

Notice the pattern: the derivative at $x = a$ seems to always equal $2a$ . This suggests that the derivative function is:

f'(x) = 2x

This is a complete function in its own right! You can evaluate it at any point:

x	f(x) = x²	f'(x) = 2x	Interpretation
-2	4	-4	Tangent slopes down (steep)
-1	1	-2	Tangent slopes down
0	0	0	Horizontal tangent (minimum!)
1	1	2	Tangent slopes up
2	4	4	Tangent slopes up (steep)

What the Derivative Function Tells Us

The derivative function $f'(x) = 2x$ encodes all the slope information about the parabola $f(x) = x^2$ :

When f'(x) < 0

The tangent line slopes downward.
The function is decreasing.
(For x < 0)

When f'(x) = 0

The tangent line is horizontal.
Potential maximum or minimum.
(At x = 0)

When f'(x) > 0

The tangent line slopes upward.
The function is increasing.
(For x > 0)

Formal Definition: The Derivative Function

We now extend our definition from a single point to all points where the limit exists:

Definition: The Derivative Function

The derivative of $f$ is the function $f'$ whose value at $x$ is:

f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}

The domain of $f'$ consists of all values of $x$ for which this limit exists.

Understanding the Definition

Note the key difference from the previous section:

Expression	Type	Meaning
f'(a)	Number	The slope at the specific point x = a
f'(x)	Function	A rule that gives the slope at any point x

The variable $x$ in $f'(x)$ is a "free variable" — it can take any value in the domain of $f'$ .

When Does f'(x) Exist?

The derivative function $f'$ may have a smaller domain than the original function $f$ . Points where $f'$ doesn't exist include:

Corners and cusps (sharp turns)
Vertical tangents
Discontinuities in $f$

We'll explore these in detail in the section on differentiability.

Interactive: The Derivative as a Function

This visualization shows a function and its derivative simultaneously. The key insight: the y-value on the derivative graph equals the slope of the tangent line on the original graph at each point.

Interactive: The Derivative as a Function

The original function (top) and its derivative function (bottom) are displayed simultaneously. Watch how the tangent line slope on the original function equals the y-value on the derivative graph at each point.

Trace x: 0.00

Show tangent line

x value

0.000

f(x) = 0.0000

0.0000

f'(x) = slope of tangent

0.0000

Key Insight

The derivative function f'(x) gives the slope of the tangent line to f(x) at every point x. Notice how:

When f(x) is increasing, f'(x) is positive
When f(x) is decreasing, f'(x) is negative
When f(x) has a horizontal tangent, f'(x) = 0

Notation Systems for the Derivative

Different mathematical traditions have developed different notations for the derivative. You'll encounter all of these, so it's important to recognize them as equivalent.

Lagrange Notation (Prime Notation)

The most common notation in calculus courses. Named after Joseph-Louis Lagrange:

First derivative: $f'(x)$ or $y'$

Second derivative: $f''(x)$ or $y''$

nth derivative: $f^{(n)}(x)$

Leibniz Notation

Developed by Gottfried Leibniz. Emphasizes the ratio of infinitesimal changes:

First derivative: $\frac{dy}{dx}$ or $\frac{df}{dx}$

Second derivative: $\frac{d^2y}{dx^2}$

Derivative operator: $\frac{d}{dx}[f(x)]$

Leibniz Notation Advantage

Leibniz notation is especially useful for the chain rule and integration by substitution, as the "dx" terms can be manipulated algebraically (with appropriate care).

Newton's Notation (Dot Notation)

Used primarily in physics, especially for time derivatives:

First derivative: $\dot{y}$ (with respect to time)

Second derivative: $\ddot{y}$

Operator Notation

Treats differentiation as an operator $D$ acting on functions:

$D f = f'$

$D^2 f = f''$

Notation	Example	Best Used For
f'(x)	f'(2) = 4	General calculus, evaluating at points
dy/dx	dy/dx = 2x	Chain rule, implicit differentiation
ẋ (dot)	ẋ = velocity	Physics (time derivatives)
Df	D(x²) = 2x	Abstract/operator theory

Computing Derivative Functions from the Definition

Let's derive some derivative functions using the limit definition. This builds intuition for the rules we'll learn later.

Example 1: f(x) = x\u00B3

Step 1: Set up the limit definition

$f'(x) = \lim_{h \to 0} \frac{(x+h)^3 - x^3}{h}$

Step 2: Expand $(x+h)^3 = x^3 + 3x^2h + 3xh^2 + h^3$

$= \lim_{h \to 0} \frac{x^3 + 3x^2h + 3xh^2 + h^3 - x^3}{h}$

Step 3: Simplify

$= \lim_{h \to 0} \frac{3x^2h + 3xh^2 + h^3}{h} = \lim_{h \to 0} (3x^2 + 3xh + h^2)$

Step 4: Take the limit as $h \to 0$

$= 3x^2$

Result: If

f(x) = x^3

, then

f'(x) = 3x^2

Example 2: f(x) = \u221Ax

Step 1: Set up the limit

$f'(x) = \lim_{h \to 0} \frac{\sqrt{x+h} - \sqrt{x}}{h}$

Step 2: Rationalize the numerator (multiply by conjugate)

$= \lim_{h \to 0} \frac{(\sqrt{x+h} - \sqrt{x})(\sqrt{x+h} + \sqrt{x})}{h(\sqrt{x+h} + \sqrt{x})}$

Step 3: Use difference of squares: $(\sqrt{a} - \sqrt{b})(\sqrt{a} + \sqrt{b}) = a - b$

$= \lim_{h \to 0} \frac{(x+h) - x}{h(\sqrt{x+h} + \sqrt{x})} = \lim_{h \to 0} \frac{h}{h(\sqrt{x+h} + \sqrt{x})}$

Step 4: Cancel $h$ and take the limit

$= \lim_{h \to 0} \frac{1}{\sqrt{x+h} + \sqrt{x}} = \frac{1}{2\sqrt{x}}$

Result: If

f(x) = \sqrt{x}

, then

f'(x) = \frac{1}{2\sqrt{x}}

(for

x > 0

)

Pattern Emerging: The Power Rule

Notice that $x^3 \to 3x^2$ and $x^{1/2} \to \frac{1}{2}x^{-1/2}$ . Both follow the pattern:

\frac{d}{dx}[x^n] = nx^{n-1}

This is the power rule, which we'll prove in the next section!

Sign Analysis: Connecting Derivatives to Behavior

The sign of the derivative tells us whether the function is increasing or decreasing. This connection is fundamental to understanding function behavior.

Interactive: Derivative Sign Analysis

Explore how the sign of the derivative determines whether the function is increasing or decreasing. Hover over the graph to see the tangent line and derivative value at any point.

The Sign Rule

The derivative acts like a "slope detector" for the original function:

f'(x) > 0 means the tangent line slopes upward → f is increasing
f'(x) < 0 means the tangent line slopes downward → f is decreasing
f'(x) = 0 means horizontal tangent → potential maximum or minimum

The Increasing/Decreasing Test

Theorem: Increasing/Decreasing Test

Let $f$ be differentiable on an interval $(a, b)$ .

If $f'(x) > 0$ for all $x$ in $(a, b)$ , then $f$ is increasing on $(a, b)$ .
If $f'(x) < 0$ for all $x$ in $(a, b)$ , then $f$ is decreasing on $(a, b)$ .
If $f'(x) = 0$ for all $x$ in $(a, b)$ , then $f$ is constant on $(a, b)$ .

Finding Critical Points

A critical point of $f$ is a value $c$ in the domain where either:

$f'(c) = 0$ (horizontal tangent), or
$f'(c)$ does not exist (corner, cusp, vertical tangent)

Critical points are where the function might change from increasing to decreasing (or vice versa) — potential local maxima and minima.

Graphing Derivatives from Original Functions

Given the graph of $f(x)$ , we can sketch $f'(x)$ by analyzing slopes:

Feature of f(x)	What it tells us about f'(x)
f is increasing	f' is positive (above x-axis)
f is decreasing	f' is negative (below x-axis)
f has a horizontal tangent	f' = 0 (crosses or touches x-axis)
f is steep (large slope)	f' has large absolute value
f has constant slope	f' is constant (horizontal)
f is linear	f' is a constant function
f is concave up	f' is increasing
f is concave down	f' is decreasing

Example: Sketching the Derivative

Consider a function that starts increasing slowly, speeds up, then slows down and levels off. What does its derivative look like?

Starts increasing slowly → f'(x) is small and positive
Speeds up (steeper) → f'(x) is increasing (getting larger positive)
Maximum steepness reached → f'(x) reaches its maximum
Slows down → f'(x) is decreasing (still positive, but smaller)
Levels off → f'(x) approaches 0

This describes a sigmoid function (like logistic growth), whose derivative is bell-shaped!

Applications: Where Derivative Functions Appear

Physics: Velocity and Acceleration Functions

If $s(t)$ is the position function, then:

Velocity Function

v(t) = s'(t) = \frac{ds}{dt}

The velocity function gives speed and direction at any time

Acceleration Function

a(t) = v'(t) = s''(t) = \frac{d^2s}{dt^2}

The acceleration function gives how velocity is changing

Economics: Marginal Functions

If $C(x)$ is the total cost of producing $x$ units:

Marginal Cost

C'(x) = \text{marginal cost}

Approximates the cost of producing one more unit when already producing

x

units

Similarly, $R'(x)$ is marginal revenue and $P'(x)$ is marginal profit.

Machine Learning Connection: The Gradient Function

In machine learning, we work with loss functions $L(\theta)$ that measure prediction error. The derivative $L'(\theta)$ (or gradient $\nabla L$ in higher dimensions) tells us how to adjust parameters to reduce loss.

The Derivative Function Powers AI

Training a neural network means computing $\frac{\partial L}{\partial \theta}$ for every parameter $\theta$ . The derivative function tells us:

Which direction increases the loss (we go the opposite way)
How sensitive the loss is to each parameter
When we've reached a minimum (gradient = 0)

Modern deep learning libraries (PyTorch, TensorFlow) compute derivative functions automatically using automatic differentiation.

Python Implementation

Numerical Derivative Function

This code demonstrates how the derivative transforms one function into another, and how to analyze function behavior using the derivative:

Numerical Derivative as a Function

🐍derivative_function.py

Explanation(6)

Code(80)

3Creating a Derivative Function

This function takes any function f and returns a NEW function f_prime that computes the derivative at any point. This is the key idea: the derivative transforms one function into another.

13The Derivative as Output

The returned function f_prime can be called with any x value to get the slope of the original function at that point. f_prime(2) gives the derivative at x=2.

17Original Function

Our test function f(x) = x³ - 3x is a cubic with interesting behavior: it has a local maximum and minimum where f'(x) = 0.

21Function Transformation

numerical_derivative(f) returns f_prime, a completely new function. This demonstrates that the derivative operator D takes functions to functions.

39Finding Critical Points

Critical points occur where f'(x) = 0. For our function, f'(x) = 3x² - 3 = 0 gives x = ±1. These are where the function has horizontal tangents.

54Sign Analysis Visualization

Shading shows where f'(x) is positive (green, f increasing) and negative (red, f decreasing). This visual makes the derivative's meaning concrete.

74 lines without explanation

1import numpy as np
2import matplotlib.pyplot as plt
3
4def numerical_derivative(f, h=1e-8):
5    """
6    Returns a function that computes the numerical derivative of f.
7
8    This transforms f(x) into f'(x) - a new function!
9    The derivative function outputs the slope of the original
10    at each input point.
11    """
12    def f_prime(x):
13        return (f(x + h) - f(x)) / h
14    return f_prime
15
16# Original function: f(x) = x^3 - 3x
17def f(x):
18    return x**3 - 3*x
19
20# Create the derivative function
21f_prime = numerical_derivative(f)
22
23# Generate x values
24x = np.linspace(-3, 3, 300)
25
26# Compute both functions at all x values
27y_original = f(x)
28y_derivative = np.array([f_prime(xi) for xi in x])
29
30# Plot both functions
31fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 8), sharex=True)
32
33# Original function
34ax1.plot(x, y_original, 'b-', linewidth=2, label='f(x) = x³ - 3x')
35ax1.axhline(y=0, color='gray', linestyle='-', alpha=0.3)
36ax1.axvline(x=0, color='gray', linestyle='-', alpha=0.3)
37ax1.set_ylabel('f(x)', fontsize=12)
38ax1.legend(loc='upper left')
39ax1.set_title('Original Function', fontsize=14)
40ax1.grid(True, alpha=0.3)
41
42# Mark critical points (where f'(x) = 0)
43critical_x = [-1, 1]
44critical_y = [f(xi) for xi in critical_x]
45ax1.scatter(critical_x, critical_y, c='red', s=100, zorder=5)
46ax1.annotate('Local Max', (-1, f(-1)), xytext=(-2, f(-1)+1),
47             arrowprops=dict(arrowstyle='->', color='red'))
48ax1.annotate('Local Min', (1, f(1)), xytext=(2, f(1)-1),
49             arrowprops=dict(arrowstyle='->', color='red'))
50
51# Derivative function
52ax2.plot(x, y_derivative, 'r-', linewidth=2, label="f'(x) = 3x² - 3")
53ax2.axhline(y=0, color='gray', linestyle='-', alpha=0.3)
54ax2.axvline(x=0, color='gray', linestyle='-', alpha=0.3)
55ax2.set_xlabel('x', fontsize=12)
56ax2.set_ylabel("f'(x)", fontsize=12)
57ax2.legend(loc='upper left')
58ax2.set_title('Derivative Function (Slope at Each Point)', fontsize=14)
59ax2.grid(True, alpha=0.3)
60
61# Shade regions based on sign of derivative
62ax2.fill_between(x, y_derivative, 0, where=(y_derivative > 0),
63                 color='green', alpha=0.3, label='f increasing')
64ax2.fill_between(x, y_derivative, 0, where=(y_derivative < 0),
65                 color='red', alpha=0.3, label='f decreasing')
66
67# Mark zeros of derivative
68ax2.scatter(critical_x, [0, 0], c='purple', s=100, zorder=5)
69
70plt.tight_layout()
71plt.show()
72
73# Print analysis
74print("Derivative Sign Analysis:")
75print("-" * 40)
76print("x < -1:  f'(x) > 0  →  f is INCREASING")
77print("x = -1:  f'(x) = 0  →  Horizontal tangent (LOCAL MAX)")
78print("-1 < x < 1: f'(x) < 0 → f is DECREASING")
79print("x = 1:   f'(x) = 0  →  Horizontal tangent (LOCAL MIN)")
80print("x > 1:   f'(x) > 0  →  f is INCREASING")

Symbolic Derivatives with SymPy

For exact symbolic computation:

Symbolic Derivative Computation

🐍symbolic_derivative.py

Explanation(3)

Code(40)

4Symbolic Mathematics

SymPy lets us work with symbolic expressions. The variable x is a symbol, not a number, allowing exact derivative computation.

20sp.diff() Computes Derivatives

sp.diff(f, x) computes the derivative of f with respect to x symbolically. This gives the exact derivative function, not a numerical approximation.

30Solving f'(x) = 0

sp.solve() finds exact solutions. For f'(x) = 3x² - 3 = 0, it returns x = -1 and x = 1, the critical points.

37 lines without explanation

1import sympy as sp
2
3# Define symbolic variable
4x = sp.Symbol('x')
5
6# Define various functions
7functions = {
8    'x²': x**2,
9    'x³': x**3,
10    'sin(x)': sp.sin(x),
11    'cos(x)': sp.cos(x),
12    'eˣ': sp.exp(x),
13    'ln(x)': sp.ln(x),
14    'x³ - 3x': x**3 - 3*x,
15}
16
17print("Function → Derivative Function")
18print("=" * 50)
19
20for name, f in functions.items():
21    # Compute the derivative symbolically
22    f_prime = sp.diff(f, x)
23    print(f"f(x) = {name:12}  →  f'(x) = {f_prime}")
24
25# Example: Finding where derivative is zero (critical points)
26print("\n" + "=" * 50)
27print("Finding Critical Points for f(x) = x³ - 3x")
28print("=" * 50)
29
30f = x**3 - 3*x
31f_prime = sp.diff(f, x)
32print(f"f'(x) = {f_prime}")
33
34# Solve f'(x) = 0
35critical_points = sp.solve(f_prime, x)
36print(f"f'(x) = 0 when x = {critical_points}")
37
38for cp in critical_points:
39    f_val = f.subs(x, cp)
40    print(f"  At x = {cp}: f({cp}) = {f_val}")

Common Mistakes to Avoid

Mistake 1: Confusing f(x) and f'(x)

$f(x)$ and $f'(x)$ are different functions. They generally have different values, different domains, and different behaviors.

Exception: $f(x) = e^x$ where $f'(x) = e^x = f(x)$ .

Mistake 2: Assuming f' Has the Same Domain as f

The derivative may not exist at some points where the original function is defined. Example: $f(x) = |x|$ is defined at $x = 0$ , but $f'(0)$ does not exist.

Mistake 3: Thinking f'(x) = 0 Always Means Extremum

A critical point where $f'(x) = 0$ might be a maximum, minimum, or neither (inflection point).

Example: $f(x) = x^3$ has $f'(0) = 0$ , but $x = 0$ is not a max or min.

Test Your Understanding

Score: 0/8

Question 1 of 8

If f(x) = x³, what is f'(x)?

Summary

The derivative is not just a number at one point — it's a function that maps inputs to slopes.

Key Concepts

Concept	Description
Derivative function f'(x)	A new function that outputs slopes of f at each point
Definition	f'(x) = lim_{h→0} [f(x+h) - f(x)]/h for all x where this exists
Sign of f'(x)	Positive → f increasing; Negative → f decreasing
f'(x) = 0	Horizontal tangent; potential max/min
Critical points	Where f'(x) = 0 or f'(x) undefined

Key Takeaways

The derivative transforms functions to functions: $f \to f'$
$f'(x)$ gives the slope of f at any point x
Multiple equivalent notations exist: $f'(x)$ , $\frac{dy}{dx}$ , $Df$
The sign of f' determines increasing/decreasing behavior
Critical points are where f'(x) = 0 or undefined
In ML, the gradient function guides parameter optimization

The Essence:

"The derivative function encodes the complete slope information of the original function — telling us at every point whether the function is climbing, descending, or momentarily level."

Coming Next: In the next section, we'll examine the formal definition of the derivative more closely, seeing how the limit of the difference quotient rigorously captures instantaneous rate of change.