The Quotient Rule

Learning Objectives

By the end of this section, you will be able to:

1. State and apply the quotient rule for differentiating quotients of functions
2. Derive the quotient rule from the product rule and chain rule
3. Prove the quotient rule from the limit definition of the derivative
4. Recognize when to use the quotient rule versus simpler alternatives
5. Connect the quotient rule to normalization operations in neural networks (softmax, layer normalization)
6. Avoid common mistakes involving sign errors and order of terms

The Big Picture: Differentiating Ratios

"When one changing quantity is divided by another, the rate of change of their ratio depends on both the top and bottom — and the bottom 'fights back' when it changes."

In the previous section, we learned the product rule for differentiating $f(x) \cdot g(x)$. But what if we need to differentiate a quotient of two functions, like $\frac{f(x)}{g(x)}$?

Just as with the product rule, a natural first guess might be that the derivative of a quotient is the quotient of derivatives: $\left(\frac{f}{g}\right)' = \frac{f'}{g'}$. But this is wrong!

The correct formula is the quotient rule:

Historical Context

The quotient rule, like the product rule, was developed by Isaac Newton and Gottfried Wilhelm Leibniz in the late 17th century. Leibniz's differential notation made the relationship between the product and quotient rules particularly clear.

Leibniz observed that division is just multiplication by a reciprocal:

This insight means the quotient rule can be derived from the product rule and chain rule — we'll do this shortly!

Intuitive Understanding: The Denominator "Fights Back"

To understand the quotient rule intuitively, imagine you're computing a ratio like miles per hour (speed).

Let $f(t)$ = miles traveled and $g(t)$ = hours elapsed. Your average speed is $v(t) = f(t)/g(t)$.

How does your speed change over time?

• When you travel more miles ($f$ increases), your speed goes up
• When more time passes ($g$ increases), your speed goes down — the denominator "dilutes" the numerator

The quotient rule captures both effects. Notice the minus sign: when the denominator increases, itdecreases the ratio. This is why we have $f' \cdot g - f \cdot g'$ in the numerator, not $f' \cdot g + f \cdot g'$.

Deriving the Quotient Rule from the Product Rule

One of the most elegant aspects of the quotient rule is that we canderive it from the product rule. The key insight is:

Use the interactive demonstration below to see each step of the derivation:

Formal Proof from the Limit Definition

Here's the rigorous proof using the limit definition of the derivative, which doesn't require the chain rule:

Proof:

Interactive Exploration

Use the visualizer below to explore the quotient rule with different function pairs. Watch how the derivative of the quotient relates to the individual functions and their derivatives.

Worked Examples

Example 1: Simple Rational Function

Example 2: The Derivative of tan(x)

Example 3: Complex Rational Function

Example 4: With Exponential Function

Special Cases and Shortcuts

When the Numerator is Constant

For $h(x) = \frac{c}{g(x)}$ where $c$ is a constant:

Alternative: Rewrite as $c \cdot g(x)^{-1}$ and use the power rule with the chain rule:

$\frac{d}{dx}\left(\frac{c}{g(x)}\right) = c \cdot (-1) \cdot g(x)^{-2} \cdot g'(x) = -\frac{c \cdot g'(x)}{[g(x)]^2}$

When to Avoid the Quotient Rule

Sometimes rewriting the expression makes differentiation easier:

Machine Learning Applications

The quotient rule appears frequently in machine learning whenever we work with normalized or ratio-based quantities.

Softmax Function

The softmax function converts logits into probabilities:

This is a quotient! When computing gradients for backpropagation, the quotient rule tells us:

Attention Mechanism

In transformer attention, the attention weights are computed as:

$\alpha_{ij} = \frac{\exp(s_{ij})}{\sum_k \exp(s_{ik})}$

This is exactly the softmax function! Backpropagating through attention requires the quotient rule to compute gradients with respect to the scores $s_{ij}$.

Layer Normalization

Layer normalization involves dividing by the standard deviation:

$\hat{x}_i = \frac{x_i - \mu}{\sigma}$

When $\sigma$ depends on $x_i$ (as it does in practice), computing the gradient requires the quotient rule.

Python Implementation

Numerical Verification

Let's verify the quotient rule numerically:

1import numpy as np
2
3def quotient_rule_numerical(f, g, x, h=1e-7):
4    """
5    Verify the quotient rule numerically.
6
7    For h(x) = f(x) / g(x), the quotient rule states:
8    h'(x) = (f'(x) * g(x) - f(x) * g'(x)) / [g(x)]^2
9
10    We'll compute both sides and compare.
11    """
12    # Numerical derivatives using central difference
13    f_prime = (f(x + h) - f(x - h)) / (2 * h)
14    g_prime = (g(x + h) - g(x - h)) / (2 * h)
15
16    # Quotient function
17    h_func = lambda t: f(t) / g(t)
18    h_prime_numerical = (h_func(x + h) - h_func(x - h)) / (2 * h)
19
20    # Quotient rule formula
21    g_x = g(x)
22    f_x = f(x)
23    h_prime_formula = (f_prime * g_x - f_x * g_prime) / (g_x ** 2)
24
25    return {
26        "f(x)": f_x,
27        "g(x)": g_x,
28        "f'(x)": f_prime,
29        "g'(x)": g_prime,
30        "h'(x) numerical": h_prime_numerical,
31        "h'(x) formula": h_prime_formula,
32        "difference": abs(h_prime_numerical - h_prime_formula)
33    }
34
35# Example 1: f(x) = x^2, g(x) = x + 1 => h(x) = x^2 / (x + 1)
36f1 = lambda x: x**2
37g1 = lambda x: x + 1
38
39result1 = quotient_rule_numerical(f1, g1, x=2.0)
40print("Example: h(x) = x^2 / (x + 1) at x = 2")
41for key, value in result1.items():
42    print(f"  {key}: {value:.6f}")
43print()
44
45# Example 2: tan(x) = sin(x) / cos(x)
46f2 = lambda x: np.sin(x)
47g2 = lambda x: np.cos(x)
48
49result2 = quotient_rule_numerical(f2, g2, x=np.pi/4)
50print("Example: h(x) = sin(x) / cos(x) = tan(x) at x = pi/4")
51for key, value in result2.items():
52    print(f"  {key}: {value:.6f}")
53
54# At x = pi/4: tan'(x) = sec^2(x) = 1/cos^2(x) = 2
55print(f"Expected sec^2(pi/4) = {1 / np.cos(np.pi/4)**2:.6f}")

Softmax Jacobian: Quotient Rule in Action

Here's how the quotient rule appears when computing the Jacobian of the softmax function:

1import numpy as np
2
3def softmax(z):
4    """
5    Softmax function: transforms logits into probabilities.
6    softmax(z)_i = exp(z_i) / sum(exp(z_j))
7
8    This is a quotient! The quotient rule appears in its derivative.
9    """
10    exp_z = np.exp(z - np.max(z))  # Numerical stability
11    return exp_z / np.sum(exp_z)
12
13def softmax_jacobian(z):
14    """
15    Compute the Jacobian of softmax: d(softmax_i)/d(z_j)
16
17    Using the quotient rule:
18    - When i = j: d(p_i)/d(z_i) = p_i(1 - p_i)
19    - When i != j: d(p_i)/d(z_j) = -p_i * p_j
20
21    This gives: Jacobian = diag(p) - p @ p.T
22    """
23    p = softmax(z)
24    n = len(z)
25    jacobian = np.zeros((n, n))
26
27    for i in range(n):
28        for j in range(n):
29            if i == j:
30                # Quotient rule: (e^z_i * S - e^z_i * e^z_i) / S^2
31                # = p_i - p_i^2 = p_i(1 - p_i)
32                jacobian[i, j] = p[i] * (1 - p[i])
33            else:
34                # Quotient rule: (0 * S - e^z_i * e^z_j) / S^2
35                # = -p_i * p_j
36                jacobian[i, j] = -p[i] * p[j]
37
38    return jacobian
39
40# Example with 3 classes
41z = np.array([2.0, 1.0, 0.1])
42p = softmax(z)
43J = softmax_jacobian(z)
44
45print("Logits z:", z)
46print("Softmax p:", p.round(4))
47print("Sum of probabilities:", p.sum())  # Should be 1
48print()
49print("Jacobian matrix:")
50print(J.round(4))
51print()
52print("Row sums (should be 0):", J.sum(axis=1).round(10))
53
54# The quotient rule tells us how changing one logit
55# affects all the probabilities!

Common Mistakes to Avoid

Test Your Understanding

Summary

The quotient rule tells us how to differentiate ratios of functions. Unlike the naive guess $f'/g'$, the correct formula accounts for both the changing numerator and the "fighting back" of the denominator.

Key Formula

Key Concepts

Key Takeaways

1. The quotient rule is not symmetric — the order matters: $f'g - fg'$, not $fg' - f'g$
2. It can be derived from the product rule by writing $f/g = f \cdot g^{-1}$
3. The squared denominator appears because we're dividing by the changing denominator
4. The quotient rule is essential for computing gradients through softmax and other normalization operations in ML
5. When the numerator is constant, rewrite with negative exponents for easier differentiation

Coming Next: In the next section, we'll learn the Chain Rule — how to differentiate compositions of functions. This is perhaps the most powerful differentiation rule and is the foundation of backpropagation in neural networks!