The Formal Definition: Limit of Difference Quotient

Learning Objectives

By the end of this section, you will be able to:

1. State the formal definition of the derivative as a limit of the difference quotient
2. Explain why we need limits to define instantaneous rate of change (the 0/0 problem)
3. Interpret the difference quotient geometrically as the slope of a secant line
4. Visualize how secant lines approach the tangent line as $h \to 0$
5. Compute derivatives from the definition for polynomial and simple functions
6. Apply the limit definition to real-world rate-of-change problems
7. Connect this definition to numerical differentiation and automatic differentiation in machine learning

The Big Picture: Capturing Instantaneous Change

"The derivative is the mathematical answer to an ancient question: How fast is something changing at this very instant?"

Imagine a ball thrown into the air. At any given instant, the ball has a definite velocity — it's moving at some specific speed in some direction. But here's the paradox that puzzled mathematicians for centuries:

The resolution to this paradox is one of the most profound ideas in mathematics: we don't compute the ratio at the instant, but rather we take the limit of ratios over shorter and shorter time intervals approaching the instant.

Historical Context: Newton, Leibniz, and the Birth of Calculus

The formal definition of the derivative emerged in the 17th century through the independent work of Isaac Newton (England) and Gottfried Wilhelm Leibniz (Germany). Both were trying to solve the same fundamental problems:

Both approaches captured the same mathematical concept, but it took another 150 years for mathematicians (notably Augustin-Louis Cauchy in the 1820s) to put the idea on rigorous footing using limits.

Building Intuition: From Average to Instantaneous

Let's trace the logical path from average rate of change to the derivative. Consider a function $f(x)$ and two points on its graph:

The Key Insight: Let h Approach Zero

Now imagine what happens as we make $h$ smaller and smaller:

1. Point Q slides along the curve toward point P
2. The secant line rotates, getting closer to the tangent line at P
3. The difference quotient approaches the slope of the tangent line

If we take the limit as $h \to 0$, we get the instantaneous rate of change — the derivative:

The Formal Definition: Limit of the Difference Quotient

Dissecting the Definition

Let's understand each component of this definition:

Equivalent Forms of the Definition

The definition can be written in several equivalent ways. Each illuminates a different aspect:

Interactive Visualization: Watching the Limit

The visualization below shows the difference quotient in action. As you decrease $h$, watch how:

• The secant line (blue) rotates toward the tangent line (green dashed)
• The difference quotient converges to the exact derivative value
• The approximation error shrinks toward zero

Understanding the Limit Process

The limit $\lim_{h \to 0}$ doesn't mean we substitute $h = 0$. That would give $\frac{0}{0}$, which is undefined. Instead, the limit asks:

"What value does the expression approach as h gets closer and closer to 0?"

The Epsilon-Delta Intuition

Formally, the limit exists and equals $L$ if we can make the difference quotient arbitrarily close to $L$ by making $h$ sufficiently small (but not zero).

When Does the Limit Fail to Exist?

The derivative may not exist at points where:

Computing Derivatives from the Definition

Let's work through several examples to see how the limit process eliminates the 0/0 problem and yields the derivative.

Example 1: f(x) = x² (The Parabola)

Example 2: f(x) = 1/x (The Hyperbola)

Example 3: f(x) = sin(x) at x = 0

Real-World Applications

The limit definition of the derivative appears throughout science and engineering whenever we need to understand instantaneous rates of change.

Physics: Instantaneous Velocity and Acceleration

Economics: Marginal Analysis

Biology: Population Growth Rate

Machine Learning Connection

The limit definition of the derivative is fundamental to how machine learning works. Every time a neural network "learns," it's using derivatives.

Numerical Gradients for Debugging

In practice, we often use the limit definition directly to verify that automatic differentiation is working correctly. This is called a gradient check:

Python Implementation

Investigating the Limit Numerically

The following code demonstrates how the difference quotient converges to the derivative as $h \to 0$:

1import numpy as np
2import matplotlib.pyplot as plt
3
4def difference_quotient(f, x, h):
5    """
6    Computes the difference quotient: [f(x+h) - f(x)] / h
7    This approximates the derivative as h → 0.
8    """
9    return (f(x + h) - f(x)) / h
10
11def investigate_limit(f, x, df_exact):
12    """
13    Shows how the difference quotient converges to the derivative
14    as h approaches 0 from different directions.
15    """
16    h_values = [1, 0.5, 0.1, 0.05, 0.01, 0.001, 0.0001, 0.00001]
17
18    print(f"Investigating f'({x}) using limit definition")
19    print(f"True derivative: {df_exact}")
20    print("-" * 55)
21    print(f"{'h':>12} | {'[f(x+h)-f(x)]/h':>18} | {'Error':>15}")
22    print("-" * 55)
23
24    for h in h_values:
25        approx = difference_quotient(f, x, h)
26        error = abs(approx - df_exact)
27        print(f"{h:>12.6f} | {approx:>18.10f} | {error:>15.2e}")
28
29    print("-" * 55)
30    print(f"{'h → 0':>12} | {df_exact:>18.10f} | {'0':>15}")
31
32# Example 1: f(x) = x² at x = 3
33# Derivative: f'(x) = 2x, so f'(3) = 6
34print("=" * 60)
35print("Example 1: f(x) = x² at x = 3")
36print("=" * 60)
37investigate_limit(lambda x: x**2, x=3, df_exact=6)
38
39# Example 2: f(x) = sin(x) at x = 0
40# Derivative: f'(x) = cos(x), so f'(0) = 1
41print("\n" + "=" * 60)
42print("Example 2: f(x) = sin(x) at x = 0")
43print("=" * 60)
44investigate_limit(np.sin, x=0, df_exact=1)
45
46# Example 3: f(x) = e^x at x = 1
47# Derivative: f'(x) = e^x, so f'(1) = e
48print("\n" + "=" * 60)
49print(f"Example 3: f(x) = eˣ at x = 1")
50print("=" * 60)
51investigate_limit(np.exp, x=1, df_exact=np.e)

Visualizing the Geometric Meaning

This code creates a visual demonstration of how secant lines approach the tangent line:

1import numpy as np
2import matplotlib.pyplot as plt
3from matplotlib.patches import FancyArrowPatch
4
5def visualize_secant_to_tangent(f, df, x0, title=""):
6    """
7    Visualizes how secant lines approach the tangent line
8    as h → 0. This is the geometric meaning of the derivative.
9    """
10    fig, axes = plt.subplots(1, 2, figsize=(14, 5))
11
12    # Left plot: Multiple secant lines approaching tangent
13    ax1 = axes[0]
14    x = np.linspace(x0 - 2, x0 + 2, 200)
15    y = f(x)
16
17    ax1.plot(x, y, 'k-', linewidth=2.5, label='f(x)')
18
19    # Plot secant lines for different h values
20    h_values = [1.5, 1.0, 0.5, 0.2, 0.05]
21    colors = plt.cm.Blues(np.linspace(0.3, 0.9, len(h_values)))
22
23    for h, color in zip(h_values, colors):
24        # Secant line through (x0, f(x0)) and (x0+h, f(x0+h))
25        slope = (f(x0 + h) - f(x0)) / h
26        y_secant = f(x0) + slope * (x - x0)
27        ax1.plot(x, y_secant, '-', color=color, linewidth=1.5,
28                 alpha=0.7, label=f'h = {h}')
29        ax1.plot(x0 + h, f(x0 + h), 'o', color=color, markersize=6)
30
31    # Plot tangent line (the limit)
32    slope_tangent = df(x0)
33    y_tangent = f(x0) + slope_tangent * (x - x0)
34    ax1.plot(x, y_tangent, 'g-', linewidth=2.5,
35             label=f"Tangent (h→0)", linestyle='--')
36
37    # Mark the point
38    ax1.plot(x0, f(x0), 'ro', markersize=10, zorder=5)
39    ax1.annotate(f'P = ({x0}, {f(x0):.2f})', (x0, f(x0)),
40                 xytext=(x0 - 0.5, f(x0) + 0.5),
41                 fontsize=10, ha='right')
42
43    ax1.set_xlim(x0 - 2, x0 + 2)
44    ax1.set_ylim(min(y) - 1, max(y) + 1)
45    ax1.set_xlabel('x', fontsize=12)
46    ax1.set_ylabel('y', fontsize=12)
47    ax1.set_title('Secant Lines → Tangent Line', fontsize=14)
48    ax1.legend(loc='upper left', fontsize=9)
49    ax1.grid(True, alpha=0.3)
50    ax1.axhline(y=0, color='gray', linewidth=0.5)
51    ax1.axvline(x=0, color='gray', linewidth=0.5)
52
53    # Right plot: Convergence of difference quotient
54    ax2 = axes[1]
55    h_vals = np.logspace(-6, 1, 50)
56    dq_vals = [(f(x0 + h) - f(x0)) / h for h in h_vals]
57
58    ax2.semilogx(h_vals, dq_vals, 'b-', linewidth=2,
59                 label='Difference quotient')
60    ax2.axhline(y=df(x0), color='g', linestyle='--', linewidth=2,
61                label=f"f'({x0}) = {df(x0):.4f}")
62
63    ax2.set_xlabel('h (log scale)', fontsize=12)
64    ax2.set_ylabel('[f(x+h) - f(x)] / h', fontsize=12)
65    ax2.set_title('Convergence to Derivative', fontsize=14)
66    ax2.legend(loc='upper right')
67    ax2.grid(True, alpha=0.3)
68    ax2.set_xlim(1e-6, 10)
69
70    plt.suptitle(title, fontsize=14, y=1.02)
71    plt.tight_layout()
72    plt.show()
73
74# Visualize for f(x) = x² at x = 2
75visualize_secant_to_tangent(
76    f=lambda x: x**2,
77    df=lambda x: 2*x,
78    x0=2,
79    title="The Derivative as a Limit: f(x) = x²"
80)

Numerical Differentiation: Approximating Derivatives

When we can't find derivatives symbolically, we use numerical approximations based directly on the limit definition.

Three Common Formulas

The central difference is more accurate because errors from the left and right partially cancel. This is why gradient checking in neural networks uses the central difference formula.

Choosing h: The Trade-off

Common Mistakes to Avoid

Test Your Understanding

Summary

The derivative is defined as the limit of the difference quotient, which captures the idea of instantaneous rate of change without ever dividing by zero.

The Central Equation

Key Concepts

Key Takeaways

1. The derivative captures instantaneous rate of change by taking a limit of average rates of change
2. Geometrically, the derivative is the slope of the tangent line, which is the limit of secant line slopes
3. The limit process avoids the 0/0 problem by examining what happens as we approach h = 0, not at h = 0
4. Computing derivatives from the definition requires algebraic simplification to cancel the h in the denominator
5. Numerical differentiation approximates derivatives using small but nonzero h values
6. This definition is the foundation of machine learning — every gradient computation ultimately rests on this limit

Coming Next: In the next section, we'll explore differentiability — when does the derivative exist, and what can go wrong at corners, cusps, and discontinuities?