Boo-AI — Master Artificial Intelligence by Building from Scratch

Learning Objectives

By the end of this section, you will be able to:

Distinguish between explicit and implicit definitions of functions
Apply implicit differentiation to find derivatives when y is not explicitly solved for in terms of x
Recognize how the chain rule appears in every implicit differentiation problem
Find tangent lines to curves defined by implicit equations
Compute second derivatives using implicit differentiation
Connect implicit differentiation to modern applications in computer graphics and machine learning

The Big Picture: Differentiating the Undifferentiable

"Implicit differentiation lets us find derivatives even when we can't — or don't want to — solve for y explicitly."

So far, we've differentiated functions given in explicit form: equations like $y = x^2$ or $y = \sin(x)$ where y is isolated on one side. But what about equations like:

x² + y² = 25

Circle

x³ + y³ = 6xy

Folium of Descartes

sin(xy) = y

Transcendental curve

These equations define y as a function of x, but y isn't explicitly solved for. We say y is defined implicitly by the equation. The brilliant technique of implicit differentiation lets us find dy/dx without ever solving for y!

The Key Insight

Implicit differentiation is just the chain rule in disguise. Whenever we differentiate a term containing y, we remember that y is a function of x, so we must apply the chain rule and multiply by dy/dx.

Historical Context

The technique of implicit differentiation emerged naturally from Leibniz's notation in the late 17th century. When faced with geometric curves that couldn't be expressed as y = f(x), Leibniz simply differentiated both sides of the equation and solved for dy/dx.

Today, implicit functions and their derivatives are fundamental in:

Computer graphics: Implicit surfaces define 3D shapes in ray tracing and CAD software
Physics simulations: Constraint equations in mechanical systems are often implicit
Economics: Indifference curves and isoquants are defined implicitly
Machine learning: Neural Radiance Fields (NeRF) and implicit neural representations use implicit surfaces

What is an Implicit Function?

An equation is explicit when y is isolated:

y = x² + 3x - 7(Explicit)

An equation is implicit when y appears mixed with x:

x² + y² = 25(Implicit)

Sometimes we could solve for y explicitly. For the circle, we get:

y = \pm\sqrt{25 - x^2}

But notice the ±! The circle defines two functions: the upper semicircle ( $y = \sqrt{25-x^2}$ ) and the lower semicircle ( $y = -\sqrt{25-x^2}$ ). Implicit differentiation handles both simultaneously with a single formula.

For more complex curves like the folium of Descartes, solving for y explicitly is impractical or impossible. Implicit differentiation is our only option!

The Implicit Differentiation Technique

The method is straightforward once you understand the key principle:

The Implicit Differentiation Process

Differentiate both sides of the equation with respect to x
Apply the chain rule to any term containing y, multiplying by dy/dx
Solve algebraically for dy/dx

Why the Chain Rule Appears

When we differentiate y² with respect to x, we're differentiating a composite function: y² where y = y(x). The chain rule says:

\frac{d}{dx}[y^2] = 2y \cdot \frac{dy}{dx}

The factor dy/dx appears because y depends on x

This is exactly what we learned in the chain rule section! Every term involving y gets multiplied by dy/dx.

Example: The Circle

Let's find dy/dx for $x^2 + y^2 = 25$ :

Step 1: Differentiate both sides with respect to x

\frac{d}{dx}[x^2] + \frac{d}{dx}[y^2] = \frac{d}{dx}[25]

Step 2: Apply the chain rule to y²

2x + 2y \cdot \frac{dy}{dx} = 0

Step 3: Solve for dy/dx

2y \cdot \frac{dy}{dx} = -2x

\frac{dy}{dx} = -\frac{x}{y}

The Formula Covers Both Branches

At the point (3, 4) on the upper semicircle: dy/dx = -3/4 (negative slope).

At the point (3, -4) on the lower semicircle: dy/dx = -3/(-4) = 3/4 (positive slope).

The formula dy/dx = -x/y automatically gives the correct slope for whichever branch we're on!

Step-by-Step Demonstration

Work through several examples of implicit differentiation step by step. Watch how the chain rule appears in each derivative.

Step-by-Step Implicit Differentiation

Derivation Steps (1/5)

Start with the equation

We have a circle of radius 3 centered at the origin.

x² + y² = 9

The curve x² + y² = 9 with tangent line (shown at final step)

Why the Chain Rule is Essential

When we differentiate y\u00b2 with respect to x, we're treating y as a function of x. By the chain rule:

d/dx[y\u00b2] = 2y \u00b7 dy/dx

The factor of dy/dx appears because y itself depends on x. This is the heart of implicit differentiation!

Interactive Exploration

Explore different implicit curves interactively. Drag the point along the curve and watch how the derivative changes. The visualizer shows both the tangent line and the step-by-step computation.

Interactive Implicit Differentiation

x² + y² = 4

Tangent Length:2.0

Implicit Curve

Point (drag to move)

Tangent Line

Normal Vector

Point on Curve

x = 1.0000

y = 1.7321

F(x,y) = -0.000000

Derivative

dy/dx = -0.5774

Slope angle: -30.0°

Implicit Differentiation for x² + y² = 4

Step 1: The equation defines y implicitly

F(x, y) = 0

Step 2: Differentiate both sides with respect to x

dF/dx + (dF/dy)(dy/dx) = 0

Step 3: Solve for dy/dx

dy/dx = -(dF/dx) / (dF/dy)

Step 4: Compute partial derivatives at point

F_x(1.00, 1.73) = 2.0000

F_y(1.00, 1.73) = 3.4641

Result:

dy/dx = -2.0000 / 3.4641 = -0.5774

Worked Examples

Example 1: Ellipse

Find dy/dx for the ellipse $\frac{x^2}{16} + \frac{y^2}{9} = 1$

Solution:

Differentiate both sides:

$\frac{2x}{16} + \frac{2y}{9} \cdot \frac{dy}{dx} = 0$

Simplify:

$\frac{x}{8} + \frac{2y}{9} \cdot \frac{dy}{dx} = 0$

Solve for dy/dx:

$\frac{dy}{dx} = -\frac{9x}{16y}$

Example 2: Product of x and y

Find dy/dx for $xy = 12$

Solution: We need the product rule on the left side:

$\frac{d}{dx}[xy] = x \cdot \frac{dy}{dx} + y \cdot 1 = 0$

Solve for dy/dx:

$\frac{dy}{dx} = -\frac{y}{x}$

Example 3: Transcendental Equation

Find dy/dx for $\sin(xy) = y$

Solution: The left side requires both the chain rule and product rule:

$\cos(xy) \cdot \frac{d}{dx}[xy] = \frac{dy}{dx}$

Apply product rule to xy:

$\cos(xy) \cdot \left(y + x\frac{dy}{dx}\right) = \frac{dy}{dx}$

Expand:

$y\cos(xy) + x\cos(xy)\frac{dy}{dx} = \frac{dy}{dx}$

Collect dy/dx terms:

$y\cos(xy) = \frac{dy}{dx} - x\cos(xy)\frac{dy}{dx}$

Factor and solve:

$\frac{dy}{dx} = \frac{y\cos(xy)}{1 - x\cos(xy)}$

Example 4: Power of y

Find dy/dx for the curve $y^2 = x^3$ (called a semicubical parabola)

Solution:

Differentiate:

$2y \cdot \frac{dy}{dx} = 3x^2$

Solve for dy/dx:

$\frac{dy}{dx} = \frac{3x^2}{2y}$

Finding Tangent Lines to Implicit Curves

Once we have dy/dx from implicit differentiation, finding the equation of a tangent line is straightforward:

y - y_0 = m(x - x_0)

where m = dy/dx evaluated at (x₀, y₀)

Example: Tangent to a Circle

Find the equation of the tangent line to $x^2 + y^2 = 25$ at the point (3, 4).

Solution:

We found that dy/dx = -x/y. At (3, 4):

$\left.\frac{dy}{dx}\right|_{(3,4)} = -\frac{3}{4}$

The tangent line equation:

$y - 4 = -\frac{3}{4}(x - 3)$

Simplify:

$y = -\frac{3}{4}x + \frac{25}{4}$

Verification

You can verify: the tangent line at (3, 4) on a circle centered at the origin should be perpendicular to the radius from (0, 0) to (3, 4). The radius has slope 4/3, and the tangent has slope -3/4. Indeed, (4/3)(-3/4) = -1, confirming perpendicularity!

Second Derivatives Using Implicit Differentiation

We can find the second derivative d²y/dx² by differentiating dy/dx implicitly again. The key is that dy/dx itself is a function of both x and y.

Example: Second Derivative of the Circle

For $x^2 + y^2 = 25$ , we found $\frac{dy}{dx} = -\frac{x}{y}$ . Find d²y/dx².

Solution: Differentiate dy/dx using the quotient rule:

$\frac{d^2y}{dx^2} = \frac{d}{dx}\left[-\frac{x}{y}\right] = -\frac{y(1) - x\frac{dy}{dx}}{y^2}$

Substitute dy/dx = -x/y:

$= -\frac{y - x\left(-\frac{x}{y}\right)}{y^2} = -\frac{y + \frac{x^2}{y}}{y^2}$

Simplify:

$= -\frac{\frac{y^2 + x^2}{y}}{y^2} = -\frac{y^2 + x^2}{y^3}$

Since x² + y² = 25:

$\frac{d^2y}{dx^2} = -\frac{25}{y^3}$

Interpreting the Second Derivative

At (3, 4) on the upper semicircle: d²y/dx² = -25/64 < 0, confirming the curve is concave down.

At (3, -4) on the lower semicircle: d²y/dx² = -25/(-64) = 25/64 > 0, confirming the curve is concave up.

Applications in Machine Learning

Implicit differentiation has become crucial in modern machine learning, particularly in:

1. Implicit Neural Representations (INRs)

Neural Radiance Fields (NeRF) and similar techniques represent 3D shapes as level sets of neural networks:

F_\theta(x, y, z) = 0

The surface is where the neural network output equals zero

To render these surfaces, we need gradients $\nabla F = \left(\frac{\partial F}{\partial x}, \frac{\partial F}{\partial y}, \frac{\partial F}{\partial z}\right)$ — exactly what implicit differentiation provides!

2. Differentiable Physics

Physics simulations often involve constraints like:

Conservation of energy: E(q, p) = constant
Rigid body constraints: |r₁ - r₂| = L
Contact constraints in collision detection

Differentiating through these constraints for gradient-based learning requires implicit differentiation.

3. The Implicit Function Theorem in Optimization

When training neural networks with implicit layers (like fixed-point iterations or optimization layers), backpropagation uses the implicit function theorem — a generalization of implicit differentiation to higher dimensions.

Python Implementation

Numerical Implicit Differentiation

Here's how to compute implicit derivatives numerically for any curve F(x, y) = 0:

Computing Implicit Derivatives

🐍implicit_differentiation.py

Explanation(4)

Code(68)

3Implicit Derivative Function

This function computes dy/dx for any implicit curve F(x,y) = 0 using numerical partial derivatives and the implicit differentiation formula.

12Partial Derivative with respect to x

We compute F_x using the central difference formula, treating y as constant. This gives the rate of change of F as x varies.

16Partial Derivative with respect to y

Similarly, we compute F_y treating x as constant. This tells us how F changes as y varies.

23The Implicit Differentiation Formula

The key formula: dy/dx = -F_x/F_y. This comes from differentiating F(x,y) = 0 implicitly and solving for dy/dx.

64 lines without explanation

1import numpy as np
2import matplotlib.pyplot as plt
3
4def implicit_derivative(F, x, y, h=1e-7):
5    """
6    Compute dy/dx for an implicit curve F(x, y) = 0
7    using the formula: dy/dx = -F_x / F_y
8
9    F: function of two variables (x, y)
10    x, y: point on the curve
11    h: step size for numerical derivatives
12    """
13    # Compute partial derivative with respect to x
14    # F_x = (F(x+h, y) - F(x-h, y)) / (2h)
15    F_x = (F(x + h, y) - F(x - h, y)) / (2 * h)
16
17    # Compute partial derivative with respect to y
18    # F_y = (F(x, y+h) - F(x, y-h)) / (2h)
19    F_y = (F(x, y + h) - F(x, y - h)) / (2 * h)
20
21    # Implicit differentiation formula
22    # From F(x, y) = 0, differentiate: F_x + F_y * dy/dx = 0
23    # Therefore: dy/dx = -F_x / F_y
24    if abs(F_y) < 1e-10:
25        return float('inf')  # Vertical tangent
26
27    return -F_x / F_y
28
29# Example 1: Circle x² + y² = 9
30def circle(x, y):
31    return x**2 + y**2 - 9
32
33# Test at point (3/√2, 3/√2) ≈ (2.12, 2.12)
34x1, y1 = 3/np.sqrt(2), 3/np.sqrt(2)
35slope1 = implicit_derivative(circle, x1, y1)
36
37print("Circle: x² + y² = 9")
38print(f"  Point: ({x1:.4f}, {y1:.4f})")
39print(f"  Numerical dy/dx: {slope1:.6f}")
40print(f"  Analytical dy/dx = -x/y = {-x1/y1:.6f}")
41print()
42
43# Example 2: Ellipse x²/4 + y²/9 = 1
44def ellipse(x, y):
45    return x**2/4 + y**2/9 - 1
46
47x2, y2 = 1.0, 3*np.sqrt(3)/2  # Point on ellipse
48slope2 = implicit_derivative(ellipse, x2, y2)
49
50print("Ellipse: x²/4 + y²/9 = 1")
51print(f"  Point: ({x2:.4f}, {y2:.4f})")
52print(f"  Numerical dy/dx: {slope2:.6f}")
53print(f"  Analytical dy/dx = -9x/(4y) = {-9*x2/(4*y2):.6f}")
54print()
55
56# Example 3: Folium of Descartes x³ + y³ = 3xy
57def folium(x, y):
58    return x**3 + y**3 - 3*x*y
59
60x3, y3 = 1.5, 1.5
61slope3 = implicit_derivative(folium, x3, y3)
62
63print("Folium: x³ + y³ = 3xy")
64print(f"  Point: ({x3:.4f}, {y3:.4f})")
65print(f"  Numerical dy/dx: {slope3:.6f}")
66# Analytical: dy/dx = (y - x²)/(y² - x)
67analytical3 = (y3 - x3**2)/(y3**2 - x3)
68print(f"  Analytical dy/dx = (y-x²)/(y²-x) = {analytical3:.6f}")

Implicit Neural Representations

This example shows how implicit surfaces are used in modern ML:

Implicit Surfaces in Machine Learning

🐍implicit_neural_surface.py

Explanation(3)

Code(76)

3Implicit Neural Representations

Modern ML represents 3D shapes as neural networks that output a signed distance. The surface is where F(x,y,z) = 0 — an implicit representation!

32Computing the Gradient

The gradient ∇F = (∂F/∂x, ∂F/∂y, ∂F/∂z) is computed using partial derivatives. This is 3D implicit differentiation.

44Surface Normal Computation

The normalized gradient gives the surface normal, crucial for lighting in neural rendering. This is exactly what implicit differentiation provides!

73 lines without explanation

1import numpy as np
2
3class ImplicitNeuralSurface:
4    """
5    Demonstrates implicit neural representations (INRs).
6    These are neural networks that learn to represent shapes
7    as level sets of learned functions: F(x, y, z) = 0
8
9    Modern ML uses implicit differentiation for:
10    - Neural Radiance Fields (NeRF)
11    - Signed Distance Functions (SDFs)
12    - Differentiable physics simulations
13    """
14
15    def __init__(self, hidden_size=64):
16        # Simple MLP: input (x,y,z) -> hidden -> output (signed distance)
17        self.W1 = np.random.randn(3, hidden_size) * 0.1
18        self.b1 = np.zeros(hidden_size)
19        self.W2 = np.random.randn(hidden_size, 1) * 0.1
20        self.b2 = np.zeros(1)
21
22    def forward(self, x, y, z):
23        """Compute F(x, y, z) - the signed distance function"""
24        inp = np.array([x, y, z])
25        # Hidden layer with tanh activation
26        h = np.tanh(inp @ self.W1 + self.b1)
27        # Output layer
28        out = h @ self.W2 + self.b2
29        return out[0]
30
31    def compute_gradient(self, x, y, z, h=1e-5):
32        """
33        Compute ∇F = (∂F/∂x, ∂F/∂y, ∂F/∂z)
34
35        The gradient points in the direction of steepest increase.
36        For a signed distance function, |∇F| ≈ 1, and ∇F is the
37        surface normal when F(x,y,z) = 0.
38        """
39        dFdx = (self.forward(x+h, y, z) - self.forward(x-h, y, z)) / (2*h)
40        dFdy = (self.forward(x, y+h, z) - self.forward(x, y-h, z)) / (2*h)
41        dFdz = (self.forward(x, y, z+h) - self.forward(x, y, z-h)) / (2*h)
42        return np.array([dFdx, dFdy, dFdz])
43
44    def surface_normal(self, x, y, z):
45        """
46        Compute the unit surface normal at a point.
47        This is the normalized gradient of F.
48
49        In rendering (like NeRF), normals are crucial for lighting!
50        """
51        grad = self.compute_gradient(x, y, z)
52        norm = np.linalg.norm(grad)
53        if norm < 1e-10:
54            return np.array([0., 0., 1.])
55        return grad / norm
56
57# Demonstrate with a simple example
58model = ImplicitNeuralSurface(hidden_size=32)
59
60# Sample a point and compute its gradient
61point = (0.5, 0.3, 0.2)
62F_value = model.forward(*point)
63gradient = model.compute_gradient(*point)
64normal = model.surface_normal(*point)
65
66print("Implicit Neural Representation Demo")
67print("=" * 40)
68print(f"Point: {point}")
69print(f"F(x,y,z) = {F_value:.6f}")
70print(f"∇F = {gradient}")
71print(f"Surface normal = {normal}")
72print()
73print("Key Insight:")
74print("  The gradient ∇F gives the surface normal.")
75print("  This is implicit differentiation in 3D!")
76print("  Used in NeRF, SDFs, and differentiable rendering.")

Common Mistakes to Avoid

Mistake 1: Forgetting the chain rule on y terms

Wrong: $\frac{d}{dx}[y^2] = 2y$

Correct: $\frac{d}{dx}[y^2] = 2y \cdot \frac{dy}{dx}$

Every term with y needs the chain rule because y depends on x!

Mistake 2: Forgetting to differentiate constants correctly

Wrong: $\frac{d}{dx}[25] = 25$

Correct: $\frac{d}{dx}[25] = 0$

The derivative of a constant is always zero.

Mistake 3: Algebraic errors when solving for dy/dx

Be careful when moving terms and factoring. A common error is dropping dy/dx from a term or making sign errors.

Always double-check by substituting a known point and comparing with the numerical derivative.

Mistake 4: Not using the product rule when needed

For terms like xy or x²y:

Wrong: $\frac{d}{dx}[xy] = \frac{dy}{dx}$

Correct: $\frac{d}{dx}[xy] = y + x\frac{dy}{dx}$

When x and y are multiplied, use the product rule!

Mistake 5: Division by zero

If the formula for dy/dx has y in the denominator, the derivative is undefined when y = 0. This corresponds to vertical tangent lines on the curve.

For a circle x² + y² = r², dy/dx = -x/y is undefined at (±r, 0), where the tangent is indeed vertical.

Test Your Understanding

Question 1 of 8

For the circle x² + y² = 25, what is dy/dx using implicit differentiation?

x² + y² = 25

Score: 0/0

Summary

Implicit differentiation extends our derivative toolkit to curves that can't be written as y = f(x). The technique is elegant: just differentiate both sides, apply the chain rule to y-terms, and solve for dy/dx.

Key Formula

For F(x, y) = 0:

\frac{dy}{dx} = -\frac{\partial F/\partial x}{\partial F/\partial y} = -\frac{F_x}{F_y}

This is the general implicit differentiation formula

Key Concepts

Concept	Description
Implicit equation	An equation where y is not isolated, like x² + y² = 25
Chain rule connection	Every y-term requires multiplying by dy/dx when differentiating
Product rule	Terms like xy require product rule: d/dx[xy] = y + x(dy/dx)
Solving for dy/dx	Collect all dy/dx terms on one side and factor
Undefined derivatives	When denominator is zero, the tangent is vertical
ML applications	NeRF, implicit surfaces, and differentiable physics

Key Takeaways

Implicit differentiation is the chain rule applied systematically to equations where y isn't explicitly solved for
Every term containing y gets multiplied by dy/dx when differentiating with respect to x
The result dy/dx is typically expressed in terms of both x and y — and that's fine!
Implicit differentiation gives one formula that works for all branches of the curve
Modern applications include neural implicit representations and differentiable physics simulations
Verify your answer by checking a known point or using numerical differentiation

Implicit Differentiation in One Sentence:

"Differentiate everything, remember y depends on x (chain rule!), then solve for dy/dx."

Coming Next: In the next section, we'll explore Higher-Order Derivatives — how to take derivatives of derivatives and what they tell us about acceleration, concavity, and the behavior of functions.