Boo-AI — Master Artificial Intelligence by Building from Scratch

Learning Objectives

By the end of this section, you will be able to:

Define limits of functions of two or more variables using the formal ε-δ definition
Evaluate limits using algebraic techniques, substitution, and the squeeze theorem
Prove that a limit does not exist by finding two paths with different limits
Apply polar coordinate transformations to analyze limits at the origin
Determine where a function of two variables is continuous
Recognize the connection between multivariable limits and convergence in optimization algorithms

The Big Picture: When Simple Becomes Complex

"In one dimension, there are only two directions to approach a point. In two dimensions, there are infinitely many."

In single-variable calculus, a limit $\lim_{x \to a} f(x) = L$ exists if the function approaches $L$ from both the left and the right. But when we move to functions of two variables, the situation becomes dramatically more complex.

One Variable: Two Directions

To check if $\lim_{x \to a} f(x)$ exists, we only need to verify that $f(x) \to L$ as $x \to a^-$ and $x \to a^+$ .

Two Variables: Infinite Directions

For $\lim_{(x,y) \to (a,b)} f(x,y)$ , we must check that $f$ approaches the same value along every possible path to $(a,b)$ .

This requirement makes multivariable limits both more challenging and more interesting. A function might approach the same limit along every straight line, yet fail to have a limit overall because of its behavior along curved paths!

Historical Context: The Road to Rigor

The formal theory of multivariable limits developed alongside the rigorization of calculus in the 19th century.

The Challenge of Multiple Dimensions

Augustin-Louis Cauchy (1789-1857) established the foundations of limit theory for single-variable functions. However, extending these ideas to multiple variables required new geometric intuition.

Karl Weierstrass (1815-1897) developed the ε-δ definition of limits that we use today. His formulation elegantly generalizes to any number of dimensions using the Euclidean distance.

Why Rigor Matters

Before rigorous definitions, mathematicians relied on intuition and geometric arguments. The discovery of pathological functions — continuous everywhere but differentiable nowhere, for example — showed that intuition could be misleading. Precise definitions protect us from such surprises.

Limits in Two Variables: The Definition

We begin with the intuitive idea: $f(x,y)$ approaches $L$ as $(x,y)$ approaches $(a,b)$ if we can make $f(x,y)$ as close to $L$ as we like by taking $(x,y)$ sufficiently close to $(a,b)$ .

Definition: Limit of a Function of Two Variables

Let $f$ be a function of two variables defined on some open disk containing $(a,b)$ , except possibly at $(a,b)$ itself. We say:

\lim_{(x,y) \to (a,b)} f(x,y) = L

if for every $\varepsilon > 0$ there exists a $\delta > 0$ such that:

\text{whenever } 0 < \sqrt{(x-a)^2 + (y-b)^2} < \delta, \text{ we have } |f(x,y) - L| < \varepsilon

Understanding the Definition

Symbol	Meaning	Geometric Interpretation
ε (epsilon)	Tolerance for output	Vertical band around L of half-width ε
δ (delta)	Tolerance for input	Disk of radius δ centered at (a,b)
√((x-a)² + (y-b)²)	Distance from (x,y) to (a,b)	Euclidean distance in the plane
0 < distance < δ	Punctured disk	Close to (a,b) but not equal to it

The Key Difference from Single Variable

In single-variable limits, $|x - a| < \delta$ represents an interval. In two variables, $\sqrt{(x-a)^2 + (y-b)^2} < \delta$ represents a disk. This disk contains all possible approach paths.

Computing Multivariable Limits

When a limit exists, we can often compute it using familiar techniques adapted for multiple variables.

Direct Substitution

If $f$ is continuous at $(a,b)$ (polynomials, rational functions where denominator ≠ 0, compositions of continuous functions), we can substitute directly:

Example: Find $\lim_{(x,y) \to (2,1)} \frac{x^2 + y^2}{x + y}$

Solution: Since the denominator is nonzero at $(2,1)$ :

\lim_{(x,y) \to (2,1)} \frac{x^2 + y^2}{x + y} = \frac{4 + 1}{2 + 1} = \frac{5}{3}

Algebraic Simplification

For indeterminate forms like 0/0, factor and simplify before taking the limit:

Example: Find $\lim_{(x,y) \to (0,0)} \frac{x^2 - y^2}{x - y}$

Solution: Factor the numerator:

\frac{x^2 - y^2}{x - y} = \frac{(x-y)(x+y)}{x-y} = x + y

Therefore:

\lim_{(x,y) \to (0,0)} (x + y) = 0

When Limits Fail to Exist

A limit fails to exist if the function approaches different values along different paths to the point. This is the most common way to prove a limit does not exist.

Two-Path Test for Non-Existence

If $f(x,y)$ approaches different values along two different paths to $(a,b)$ , then $\lim_{(x,y) \to (a,b)} f(x,y)$ does not exist.

Classic Example: xy/(x² + y²)

Consider the function $f(x,y) = \frac{xy}{x^2 + y^2}$ . Let's examine its behavior as we approach the origin along different paths.

Path 1: Along the x-axis (y = 0)

f(x, 0) = \frac{x \cdot 0}{x^2 + 0} = 0

Limit along x-axis: 0

Path 2: Along the line y = x

f(x, x) = \frac{x \cdot x}{x^2 + x^2} = \frac{x^2}{2x^2} = \frac{1}{2}

Limit along y = x: 1/2

Path 3: Along the line y = -x

f(x, -x) = \frac{x \cdot (-x)}{x^2 + x^2} = \frac{-x^2}{2x^2} = -\frac{1}{2}

Limit along y = -x: -1/2

Since different paths give different limits (0, 1/2, and -1/2), the limit $\lim_{(x,y) \to (0,0)} \frac{xy}{x^2 + y^2}$ does not exist.

The Path Approach Test: Interactive Exploration

Let's visualize how different paths lead to different limiting values. Watch as particles travel along various paths toward the origin, revealing the path-dependent nature of certain limits.

Loading visualization...

Checking All Lines Is Not Enough

A function may approach the same limit along every straight line, yet still fail to have a limit! The classic example is $f(x,y) = \frac{x^2 y}{x^4 + y^2}$ . Along any line $y = mx$ , the limit is 0. But along the parabola $y = x^2$ , the limit is $1/2$ .

Interactive: Explore Multiple Limit Functions

Use this 3D visualization to explore different functions and see how their limits behave along various paths. Some functions have limits that exist; others do not.

Loading 3D visualization...

The Squeeze Theorem for Two Variables

When a limit exists but is difficult to compute directly, the squeeze theorem often provides a way forward.

The Squeeze Theorem (Two Variables)

If $g(x,y) \leq f(x,y) \leq h(x,y)$ for all $(x,y)$ in some punctured disk around $(a,b)$ , and if:

\lim_{(x,y) \to (a,b)} g(x,y) = \lim_{(x,y) \to (a,b)} h(x,y) = L

then:

\lim_{(x,y) \to (a,b)} f(x,y) = L

Example: Using the Squeeze Theorem

Show that $\lim_{(x,y) \to (0,0)} (x^2 + y^2) \sin\left(\frac{1}{\sqrt{x^2+y^2}}\right) = 0$

Solution: Let $r = \sqrt{x^2 + y^2}$ . Since $|\sin(1/r)| \leq 1$ :

|r^2 \sin(1/r)| \leq r^2

Therefore: $-r^2 \leq r^2 \sin(1/r) \leq r^2$

Since $\lim_{r \to 0} (-r^2) = \lim_{r \to 0} r^2 = 0$ , by the squeeze theorem:

\lim_{(x,y) \to (0,0)} (x^2 + y^2) \sin\left(\frac{1}{\sqrt{x^2+y^2}}\right) = 0

The Polar Coordinate Method

Converting to polar coordinates is a powerful technique for limits at the origin. If we set $x = r\cos\theta$ and $y = r\sin\theta$ , then approaching the origin corresponds to $r \to 0$ .

Polar Coordinate Test

Convert $f(x,y)$ to polar form $f(r\cos\theta, r\sin\theta) = g(r,\theta)$ . Then:

If $\lim_{r \to 0} g(r,\theta)$ depends on $\theta$ , the limit does not exist
If $\lim_{r \to 0} g(r,\theta) = L$ (independent of $\theta$ ), the limit might exist and equal $L$

Example: Polar Analysis of xy/(x² + y²)

Convert to polar:

f(x,y) = \frac{xy}{x^2+y^2} = \frac{(r\cos\theta)(r\sin\theta)}{r^2} = \frac{r^2\cos\theta\sin\theta}{r^2} = \cos\theta\sin\theta = \frac{1}{2}\sin(2\theta)

The result depends on $\theta$ ! As $r \to 0$ , the "limit" varies from $-1/2$ to $1/2$ depending on the direction. Therefore, the Cartesian limit does not exist.

When Polar Works Best

The polar method is most effective for limits at the origin when the function involves expressions like $x^2 + y^2$ , $xy$ , or $x^n y^m$ . It naturally handles the radial symmetry (or lack thereof) of such expressions.

Continuity in Two Variables

Just as in single-variable calculus, continuity means "no breaks or jumps." A function is continuous at a point if the limit equals the function value.

Definition: Continuity at a Point

A function $f(x,y)$ is continuous at $(a,b)$ if:

$f(a,b)$ is defined
$\lim_{(x,y) \to (a,b)} f(x,y)$ exists
$\lim_{(x,y) \to (a,b)} f(x,y) = f(a,b)$

$f$ is continuous on a region $D$ if it is continuous at every point in $D$ .

Types of Discontinuities

Type	Description	Example
Removable	Limit exists but ≠ f(a,b) or f undefined	sin(r)/r at origin (removable by defining f(0,0)=1)
Jump	Function jumps between values across a curve	f = 1 if y > 0, f = -1 if y < 0
Essential	Limit does not exist	xy/(x² + y²) at origin

Interactive: Explore Continuous and Discontinuous Surfaces

Visualize different surfaces and observe where they are continuous and where discontinuities occur. Toggle the ε-δ view to see the formal definition in action.

Loading visualization...

Standard Continuous Functions

The following types of functions are continuous on their natural domains:

Polynomials in $x$ and $y$ : continuous everywhere
Rational functions: continuous except where denominator = 0
Root functions: continuous where the expression under the root is positive (for even roots)
Trigonometric functions: continuous on their domains
Exponential and logarithmic functions: continuous on their domains
Compositions of continuous functions: continuous

The Epsilon-Delta Definition: Making It Rigorous

The ε-δ definition provides the rigorous foundation for all our limit arguments. While intuition guides us, this definition provides certainty.

Geometric Interpretation

Saying $\lim_{(x,y) \to (a,b)} f(x,y) = L$ with the ε-δ definition means:

Challenge: Someone gives you an ε > 0 (how close to $L$ the output must be)
Response: You provide a δ > 0 (how close to $(a,b)$ the input must be)
Guarantee: All points within distance δ of $(a,b)$ get mapped within distance ε of $L$

Example: Proving a Limit with ε-δ

Prove that $\lim_{(x,y) \to (0,0)} (x^2 + y^2) = 0$

Proof: Let $\varepsilon > 0$ be given. We need to find $\delta > 0$ such that:

0 < \sqrt{x^2 + y^2} < \delta \implies |x^2 + y^2 - 0| < \varepsilon

Let $r = \sqrt{x^2 + y^2}$ . Then $x^2 + y^2 = r^2$ .

Choose $\delta = \sqrt{\varepsilon}$ . If $r < \delta$ , then:

|x^2 + y^2| = r^2 < \delta^2 = \varepsilon

Thus the limit is 0.

Real-World Applications

1. Heat Distribution

The temperature at a point on a metal plate is modeled by a function $T(x,y)$ . Continuity of $T$ means there are no sudden temperature jumps — heat flows smoothly through the material.

2. Topographical Maps

Elevation as a function of position, $h(x,y)$ , is continuous for most terrain. Discontinuities represent cliffs or vertical drops. The gradient of $h$ (which requires limits to define) gives the direction of steepest ascent.

3. Fluid Flow

In fluid dynamics, velocity fields $\mathbf{v}(x,y)$ must be continuous for physical flow. Discontinuities would imply infinite accelerations, which are physically impossible.

4. Electric Potential

The electric potential $V(x,y)$ created by a charge distribution is continuous away from point charges. Understanding limits helps analyze the behavior near boundaries and interfaces.

Machine Learning Applications

Understanding multivariable limits is essential for analyzing the behavior of loss surfaces and optimization algorithms in machine learning.

1. Loss Surface Analysis

The loss function $L(\theta_1, \theta_2, \ldots, \theta_n)$ depends on all model parameters. Understanding limits helps us analyze:

Critical points: Where does $\nabla L = 0$ ?
Saddle points: Points where the loss surface "curves up" in some directions and "curves down" in others
Boundary behavior: What happens as parameters approach infinity?

2. Gradient Descent Convergence

Gradient descent follows the path $\theta_{k+1} = \theta_k - \eta \nabla L(\theta_k)$ . The question of whether this sequence converges — and to what — is fundamentally a question about limits.

Connection: A loss function that is continuous and bounded below on a closed region must achieve a minimum there (extreme value theorem in multiple variables). This guarantees that an optimal solution exists, even though finding it may be computationally hard.

3. Regularization and Continuity

Regularization terms like $\lambda \|\theta\|^2$ ensure the loss function has good continuity properties. Without regularization, loss landscapes can have pathological behavior — extremely sharp features that make optimization unstable.

Why Smoothness Matters

Many optimization guarantees require the loss function to be not just continuous, but also differentiable with continuous derivatives (C¹ smooth). This ensures gradients don't change too abruptly, allowing predictable optimization behavior.

Python Implementation

Testing Limit Existence

Numerical Limit Testing

🐍limit_testing.py

Explanation(3)

Code(106)

3Path-based Limit Testing

We approach the limit point along a parametric curve (x(t), y(t)) where t → 0 corresponds to approaching the origin.

29Multiple Path Testing

To test if a limit exists, we check many different paths. If any two paths give different limits, the limit definitely does not exist.

59The Tricky Function

f(x,y) = x²y/(x⁴+y²) demonstrates why checking only lines is insufficient. Along any line y = mx, the limit is 0. But along the parabola y = x², the limit is 1/2!

103 lines without explanation

1import numpy as np
2import matplotlib.pyplot as plt
3from mpl_toolkits.mplot3d import Axes3D
4
5def compute_limit_along_path(f, path_func, num_points=1000):
6    """
7    Numerically estimate the limit of f(x,y) as we approach
8    (0,0) along a parametric path (x(t), y(t)).
9
10    Parameters:
11    - f: function of (x, y)
12    - path_func: returns (x, y) given parameter t
13    - num_points: number of sample points
14
15    The path should satisfy path_func(0) = (0, 0).
16    """
17    # Sample t values approaching 0
18    t_values = np.linspace(1, 1e-10, num_points)
19    limits = []
20
21    for t in t_values:
22        x, y = path_func(t)
23        try:
24            val = f(x, y)
25            if np.isfinite(val):
26                limits.append(val)
27        except:
28            pass
29
30    if len(limits) > 10:
31        return np.mean(limits[-10:])  # Average of last 10 values
32    return None
33
34def test_limit_existence(f, paths):
35    """
36    Test if limit exists by checking multiple paths.
37    If all paths give the same limit, the limit MIGHT exist.
38    If paths differ, the limit definitely does NOT exist.
39    """
40    results = {}
41    for name, path in paths.items():
42        limit = compute_limit_along_path(f, path)
43        results[name] = limit
44    return results
45
46# Define test functions
47def f1(x, y):
48    """Limit exists: sin(r)/r → 1"""
49    r = np.sqrt(x**2 + y**2)
50    if r < 1e-10:
51        return 1
52    return np.sin(r) / r
53
54def f2(x, y):
55    """Limit does NOT exist: xy/(x² + y²)"""
56    r2 = x**2 + y**2
57    if r2 < 1e-10:
58        return 0
59    return (x * y) / r2
60
61def f3(x, y):
62    """Tricky: limit is 0 along all lines, but 1/2 along y = x²"""
63    denom = x**4 + y**2
64    if denom < 1e-10:
65        return 0
66    return (x**2 * y) / denom
67
68# Define approach paths
69paths = {
70    "x-axis (y=0)": lambda t: (t, 0),
71    "y-axis (x=0)": lambda t: (0, t),
72    "line y=x": lambda t: (t, t),
73    "line y=-x": lambda t: (t, -t),
74    "line y=2x": lambda t: (t, 2*t),
75    "parabola y=x²": lambda t: (t, t**2),
76    "cubic y=x³": lambda t: (t, t**3),
77}
78
79print("=" * 60)
80print("TESTING LIMIT EXISTENCE")
81print("=" * 60)
82
83# Test f1: limit should exist
84print("\nf(x,y) = sin(√(x²+y²)) / √(x²+y²)")
85print("-" * 40)
86results = test_limit_existence(f1, paths)
87for path, limit in results.items():
88    print(f"  {path}: {limit:.6f}" if limit else f"  {path}: undefined")
89print("→ All paths give ~1.0, limit EXISTS and equals 1")
90
91# Test f2: limit should NOT exist
92print("\nf(x,y) = xy / (x² + y²)")
93print("-" * 40)
94results = test_limit_existence(f2, paths)
95for path, limit in results.items():
96    print(f"  {path}: {limit:.6f}" if limit else f"  {path}: undefined")
97print("→ Different paths give different values, limit does NOT exist")
98
99# Test f3: tricky case
100print("\nf(x,y) = x²y / (x⁴ + y²)")
101print("-" * 40)
102results = test_limit_existence(f3, paths)
103for path, limit in results.items():
104    print(f"  {path}: {limit:.6f}" if limit else f"  {path}: undefined")
105print("→ All LINES give 0, but parabola y=x² gives 0.5!")
106print("→ This proves the limit does NOT exist")

Testing Continuity

Numerical Continuity Testing

🐍continuity_testing.py

Explanation(2)

Code(68)

4Numerical Continuity Test

We test the ε-δ definition numerically: for each ε, we sample points in a δ-neighborhood and check if |f(x,y) - f(a,b)| < ε.

18Random Sampling

We sample random points in a disk around (a,b) using polar coordinates. This gives a more thorough check than just testing grid points.

66 lines without explanation

1import numpy as np
2
3def is_continuous_at(f, a, b, epsilon_values=[0.1, 0.01, 0.001]):
4    """
5    Numerically test continuity at point (a, b).
6
7    For each epsilon, we find the maximum |f(x,y) - f(a,b)|
8    within a delta-neighborhood. If this can be made
9    arbitrarily small, the function appears continuous.
10    """
11    try:
12        f_center = f(a, b)
13    except:
14        return False, "Function undefined at the point"
15
16    results = []
17    for eps in epsilon_values:
18        delta = eps  # Start with delta = epsilon
19
20        # Sample points in delta-disk
21        max_deviation = 0
22        for _ in range(1000):
23            r = np.random.uniform(0, delta)
24            theta = np.random.uniform(0, 2*np.pi)
25            x = a + r * np.cos(theta)
26            y = b + r * np.sin(theta)
27
28            try:
29                deviation = abs(f(x, y) - f_center)
30                max_deviation = max(max_deviation, deviation)
31            except:
32                pass
33
34        results.append({
35            'epsilon': eps,
36            'delta': delta,
37            'max_deviation': max_deviation,
38            'satisfies': max_deviation < eps
39        })
40
41    # Check if all tests pass
42    all_pass = all(r['satisfies'] for r in results)
43    return all_pass, results
44
45# Test continuous function
46def continuous_f(x, y):
47    return x**2 + y**2
48
49# Test discontinuous function
50def discontinuous_f(x, y):
51    if x**2 + y**2 < 0.0001:
52        return 1  # Jump at origin
53    return x**2 + y**2
54
55print("Testing Continuity at (0, 0)")
56print("=" * 50)
57
58print("\nContinuous function: f(x,y) = x² + y²")
59result, details = is_continuous_at(continuous_f, 0, 0)
60print(f"Continuous: {result}")
61for d in details:
62    print(f"  ε={d['epsilon']:.3f}, max deviation={d['max_deviation']:.6f}")
63
64print("\nDiscontinuous function: f(x,y) = x² + y² with jump at origin")
65result, details = is_continuous_at(discontinuous_f, 0, 0)
66print(f"Continuous: {result}")
67for d in details:
68    print(f"  ε={d['epsilon']:.3f}, max deviation={d['max_deviation']:.6f}")

Polar Coordinate Analysis

Symbolic Polar Analysis

🐍polar_analysis.py

Explanation(2)

Code(79)

5Polar Coordinate Method

Converting to polar coordinates (x = r cos θ, y = r sin θ) often simplifies limit analysis. If the result depends on θ, the limit DNE.

25θ-Independence Check

After taking the limit as r → 0, we check if the result contains θ. If so, different approach directions give different limits.

77 lines without explanation

1import numpy as np
2import sympy as sp
3
4def analyze_limit_polar(f_expr):
5    """
6    Analyze limit using polar coordinates.
7
8    If the limit in polar coordinates depends on θ,
9    the Cartesian limit does not exist.
10
11    If the limit is independent of θ and is finite,
12    the limit exists.
13    """
14    r, theta = sp.symbols('r theta', real=True, positive=True)
15    x = r * sp.cos(theta)
16    y = r * sp.sin(theta)
17
18    # Substitute polar coordinates
19    f_polar = f_expr.subs({'x': x, 'y': y})
20    f_simplified = sp.simplify(f_polar)
21
22    print(f"Original: f(x,y) = {f_expr}")
23    print(f"In polar: f(r,θ) = {f_simplified}")
24
25    # Try to compute limit as r → 0
26    try:
27        limit = sp.limit(f_simplified, r, 0)
28        print(f"Limit as r → 0: {limit}")
29
30        # Check if limit depends on theta
31        if theta in limit.free_symbols:
32            print("⚠ Limit depends on θ → Limit does NOT exist")
33        else:
34            print("✓ Limit is independent of θ → Limit exists")
35    except:
36        print("Could not compute symbolic limit")
37
38# Define symbolic variables
39x, y = sp.symbols('x y')
40
41print("="*60)
42print("POLAR COORDINATE ANALYSIS")
43print("="*60)
44
45# Example 1: Limit exists
46print("\nExample 1: (x² + y²) * sin(1/√(x² + y²))")
47print("-" * 40)
48analyze_limit_polar((x**2 + y**2) * sp.sin(1/sp.sqrt(x**2 + y**2)))
49
50# Example 2: Limit does not exist
51print("\nExample 2: xy / (x² + y²)")
52print("-" * 40)
53analyze_limit_polar(x*y / (x**2 + y**2))
54
55# Example 3: sin(r)/r type
56print("\nExample 3: sin(√(x² + y²)) / √(x² + y²)")
57print("-" * 40)
58analyze_limit_polar(sp.sin(sp.sqrt(x**2 + y**2)) / sp.sqrt(x**2 + y**2))
59
60# Manual polar analysis for xy/(x²+y²)
61print("\n" + "="*60)
62print("MANUAL ANALYSIS: xy/(x²+y²)")
63print("="*60)
64print("""
65Substituting x = r cos(θ), y = r sin(θ):
66
67f(r,θ) = (r cos θ)(r sin θ) / (r²cos²θ + r²sin²θ)
68       = r² cos θ sin θ / r²
69       = cos θ sin θ
70       = (1/2) sin(2θ)
71
72The limit depends on θ! Different values of θ give:
73  θ = 0:     limit = 0
74  θ = π/4:   limit = 1/2
75  θ = π/2:   limit = 0
76  θ = -π/4:  limit = -1/2
77
78Therefore, the limit does NOT exist.
79""")

Test Your Understanding

Summary

Limits and continuity in multiple dimensions extend our single-variable intuition while introducing fundamentally new challenges. The key insight is that in higher dimensions, there are infinitely many ways to approach a point.

Key Concepts

Concept	Description
ε-δ Definition	For every ε > 0, there exists δ > 0 such that the function value is within ε of L when (x,y) is within δ of (a,b)
Path Test	If two paths give different limits, the overall limit DNE
Polar Method	Convert to polar: if result depends on θ, limit DNE
Squeeze Theorem	Bound f between two functions that have the same limit
Continuity	Limit exists and equals function value

Key Takeaways

Limits in multiple dimensions require the function to approach the same value along every possible path
Finding two paths with different limits proves a limit does not exist
Checking only straight lines is not sufficient — curved paths may give different limits
The polar coordinate method is powerful for analyzing limits at the origin
The squeeze theorem helps prove limits exist when direct computation is difficult
Continuity in multiple dimensions means no "tears" or "jumps" in the surface

The Essence of Multivariable Limits:

"A limit exists only when all paths lead to the same destination — not just the roads you can see, but every winding trail and spiral staircase."

Coming Next: In the next section, we'll explore partial derivatives — rates of change in individual directions. Just as limits provide the foundation for derivatives in single-variable calculus, multivariable limits underpin partial differentiation.