Logarithmic Functions: Inverting Exponentials

Learning Objectives

After completing this section, you will be able to:

1. Define logarithms as inverse functions of exponentials and translate between logarithmic and exponential forms
2. Distinguish between common logarithms (base 10), natural logarithms (base e), and binary logarithms (base 2)
3. Apply logarithm properties (product, quotient, and power rules) to simplify expressions and solve equations
4. Use the change of base formula to convert between logarithms of different bases
5. Graph logarithmic functions and identify key features: domain, range, asymptotes, and intercepts
6. Recognize real-world applications including the Richter scale, decibels, pH, and information theory
7. Understand why logarithms are essential in machine learning for numerical stability and gradient computation

The Story of Logarithms

A Revolution in Calculation

In the early 17th century, astronomers, navigators, and scientists faced an enormous computational challenge. Calculating planetary orbits, navigating ships across oceans, and conducting scientific experiments required multiplying and dividing very large numbers—a process that was tedious, error-prone, and could take hours or even days.

In 1614, Scottish mathematician John Napier published a revolutionary discovery that would transform calculation: logarithms. His key insight was profound:

Napier's Insight: Multiplication can be converted to addition by working with exponents. If you want to multiply two numbers, add their logarithms instead—then convert back.

This single idea reduced multiplication to addition, division to subtraction, and exponentiation to simple multiplication. Before electronic calculators, logarithm tables and slide rules (which are essentially analog logarithm computers) were indispensable tools used by scientists and engineers for over 300 years.

The Name "Logarithm"

Napier coined the term from Greek: logos (ratio, proportion) and arithmos (number). A logarithm is literally a "ratio number"—it captures the ratio or proportion of growth in exponential processes.

Definition: The Inverse of Exponentials

The Fundamental Relationship

A logarithm answers a simple question: "To what power must I raise the base to get this number?"

Formally, for any base $b > 0$ with $b eq 1$:

In words: "the logarithm base $b$ of $x$" equals $y$ if and only if "$b$ to the $y$ power equals $x$."

The Inverse Function Relationship

The logarithm function $f(x) = log_b(x)$ and the exponential function $g(x) = b^x$ are inverse functions. This means they "undo" each other:

Intuition: The "What Power?" Question

Forget the symbols for a moment. Every logarithm is just a question. When you write $log_2(8)$ you are literally asking: “starting from 2, how many times must I multiply by 2 to reach 8?” The answer is 3, because $2 \cdot 2 \cdot 2 = 8$. That number you arrived at — that count of multiplications — is the logarithm.

Analogy: think of an exponential function $b^y$ as a recipe (“multiply $b$ by itself $y$ times to get a size $x$”). The logarithm is the recipe reader: given the finished cake of size $x$, it tells you how many times you must have doubled (or tripled, or $b$-tupled) to bake it. Exponentials grow; logs count growth steps. They are two views of the same coin.

Common and Natural Logarithms

The Three Most Important Bases

The Natural Logarithm: Why Base e?

The natural logarithm (base $e approx 2.71828ldots$) might seem like an arbitrary choice, but it's actually the most natural base for calculus. The number $e$ is special because:

No other base produces such clean derivatives. This is why natural logarithms dominate in calculus, differential equations, and any field dealing with continuous change.

The Natural Choice: When working with rates of change, growth, or decay, the natural logarithm is almost always the right choice. It simplifies derivatives, integrals, and the mathematics of continuous processes.

The Binary Logarithm: Counting Bits

In computer science, the binary logarithm tells you "how many bits do I need?" For a number $n$:

• $log_2(n)$ bits are needed to represent $n$ distinct values
• Binary search on $n$ elements takes $O(log_2 n)$ comparisons
• A balanced binary tree with $n$ nodes has height $log_2(n)$

Properties of Logarithms

The Three Fundamental Laws

The properties of logarithms follow directly from the properties of exponents. Since $log_b(x)$ is the inverse of $b^x$, the laws of exponents transform into laws of logarithms.

Try the interactive explorer below to see these properties in action:

Key Values to Memorize

Change of Base Formula

What if you need $log_2(100)$ but your calculator only has ln and log₁₀? The change of base formula converts between any bases:

Graphing Logarithmic Functions

The graph of $y = log_b(x)$ is the reflection of $y = b^x$ across the line $y = x$. This reflection relationship between inverse functions is the heart of logarithmic graphs — swap x and y, and you swap the two curves.

Drag along the blue curve below. The amber dashed segment shows the reflection across the line $y=x$: the moment you pick a point $(x, \log_b x)$, the mirror point $(\log_b x, x)$ on the green exponential curve falls into place automatically. That single picture is the inverse relationship.

Below is the more traditional graph with adjustable base and a click-to-probe coordinate readout — use it to read off specific values like $\log_2(8) = 3$ or $\ln(e) = 1$.

Key Features of Logarithmic Graphs

How the Base Affects the Graph

• Base > 1: The function is increasing. Larger x gives larger y.
• Larger base: The curve rises more slowly. Compare log₁₀(100) = 2 vs log₂(100) ≈ 6.64.
• Base between 0 and 1: The function is decreasing (rarely used in practice).

Transformations of Logarithmic Functions

Standard function transformations apply to logarithms:

Real-World Logarithmic Scales

Many natural phenomena span enormous ranges of values. Logarithmic scales compress these ranges into manageable numbers, making patterns visible that would be hidden on a linear scale.

Why Logarithmic Scales?

1. Compress huge ranges: The visible light spectrum covers wavelengths from 400nm to 700nm, but the full electromagnetic spectrum spans from 10⁻¹⁵m (gamma rays) to 10⁸m (radio waves)—a factor of 10²³!
2. Match human perception: Our ears perceive loudness logarithmically. A sound 10x more intense sounds about twice as loud.
3. Reveal multiplicative patterns: Exponential growth appears as a straight line on a log scale, making it easy to identify.
4. Compare relative changes: A doubling looks the same size whether it's from 10 to 20 or from 10,000 to 20,000.

The Richter Scale: A Deep Dive

The Richter magnitude $M$ of an earthquake is:

where $A$ is the measured amplitude and $A_0$ is a reference amplitude. Each unit increase in magnitude means:

• 10× more ground motion (amplitude)
• ~31.6× more energy released (since energy ∝ amplitude²)

Applications in Machine Learning

Logarithms are ubiquitous in machine learning, not as a historical curiosity, but as an essential tool for numerical stability and theoretical elegance.

Why ML Uses Log-Probabilities

Consider computing the likelihood of observing data given a model. If you have$n$ independent observations with probabilities $p_1, p_2, ldots, p_n$:

The Problem: If each $p_i approx 0.99$ and $n = 1000$:

For smaller probabilities or more samples, $L$ underflows to exactly 0 in floating-point arithmetic—catastrophic for training!

The Solution: Work with log-likelihood instead:

Products become sums, tiny numbers become manageable negative numbers, and gradients become stable.

Plain Python first — multiply, then add logs

Before we touch any framework, let's do log-likelihood by hand. The point of this snippet is not to be efficient — it is to make you feel the difference between a product and a sum of logs.

1import math
2
3# Imagine 8 independent coin flips. Our model predicts p = 0.6 for "heads".
4# Observed sequence: 1=heads, 0=tails.
5observations = [1, 1, 0, 1, 1, 0, 1, 1]
6p = 0.6
7
8# Approach A: multiply raw probabilities (the dangerous way)
9likelihood = 1.0
10for y in observations:
11    likelihood *= p if y == 1 else (1.0 - p)
12print("likelihood        =", likelihood)
13
14# Approach B: sum log-probabilities (the safe way)
15log_likelihood = 0.0
16for y in observations:
17    log_likelihood += math.log(p) if y == 1 else math.log(1.0 - p)
18print("log_likelihood    =", log_likelihood)
19
20# Sanity: exp of the log-likelihood should recover the raw likelihood.
21print("exp(log_lik)      =", math.exp(log_likelihood))

Cross-Entropy Loss

The cross-entropy loss for classification is the negative log-likelihood:

where $y$ is the true label (one-hot) and $hat{y}$ is the predicted probability distribution.

Log-Softmax for Numerical Stability

The softmax function converts logits to probabilities:

The problem: For large logits, $e^{z_i}$ overflows. The solution: Compute log-softmax directly:

Using the "log-sum-exp trick" with a shift $c = max(z)$:

We will now build the log-softmax twice. First in plain NumPy — line by line, with every value hand-traced — so you can see the math. Then we will hand it to PyTorch and watch the same numbers come out in one call. Plain Python first, framework second.

Step 1 — Pure NumPy: trace every line

1import numpy as np
2
3def log_softmax_naive(logits):
4    """Naive implementation -- WILL OVERFLOW for large logits."""
5    exp_logits = np.exp(logits)              # e^1000 = inf
6    softmax = exp_logits / np.sum(exp_logits)  # inf / inf = nan
7    return np.log(softmax)                   # log(nan) = nan
8
9def log_softmax_stable(logits):
10    """Stable log-softmax using the log-sum-exp trick."""
11    c = np.max(logits)                       # shift constant
12    shifted = logits - c                     # all entries <= 0
13    sum_exp = np.sum(np.exp(shifted))        # finite, well-behaved
14    log_sum_exp = c + np.log(sum_exp)        # add c back
15    return logits - log_sum_exp              # log of softmax
16
17# Extreme logits that crash the naive path
18logits = np.array([1000.0, 1001.0, 1002.0])
19print("naive :", log_softmax_naive(logits))
20print("stable:", log_softmax_stable(logits))

Step 2 — The same idea in PyTorch (one call)

Once you understand the trick by hand, PyTorch hides the boilerplate. torch.nn.functional.log_softmax already uses the log-sum-exp trick internally, with extra speed-ups for GPU and autograd.

1import torch
2import torch.nn.functional as F
3
4# Same extreme logits — now as a PyTorch tensor on the default device.
5logits = torch.tensor([1000.0, 1001.0, 1002.0])
6
7# log_softmax internally does max-shift + log-sum-exp, with autograd support.
8log_probs = F.log_softmax(logits, dim=-1)
9
10# Round-trip check: exp(log_probs) must sum to 1.
11probs = log_probs.exp()
12print("log_probs:", log_probs)
13print("probs    :", probs)
14print("sum(probs):", probs.sum().item())

Information Theory: Bits and Entropy

Claude Shannon's information theory uses the binary logarithm to measure information:

The entropy (average information content) of a distribution is:

Numerical Computing Considerations

Common Pitfalls

The log1p Function

When computing $log(1 + x)$ for small $x$, direct computation loses precision because $1 + x approx 1$ in floating point. The function log1p(x) computes $log(1 + x)$ accurately:

For tiny $x$, the Taylor series of $\ln(1+x)$ starts with $x - \tfrac{x^2}{2} + \tfrac{x^3}{3} - \dots$, so to first order $\ln(1+x) \approx x$. The naive computationnp.log(1 + x) first adds 1 to x, and that addition is where the precision is destroyed — before the logarithm ever runs. log1p sidesteps the bad addition by evaluating the series directly.

1import numpy as np
2
3x = 1e-17  # tiny: well below the float64 ulp at 1.0 (~2.22e-16)
4
5# Step 1 — the addition that ruins precision
6ones_plus_x = 1.0 + x
7print("1 + x stored as:", repr(ones_plus_x))
8
9# Step 2 — naive log loses every bit of x
10naive = np.log(1.0 + x)
11print("naive np.log(1 + x):", naive)
12
13# Step 3 — log1p evaluates the series, no precision loss
14accurate = np.log1p(x)
15print("np.log1p(x)        :", accurate)
16
17# Step 4 — true value to compare against
18true_value = x - 0.5 * x**2  # first two Taylor terms
19print("true ≈ x - x^2/2   :", true_value)

Summary

Key Takeaways

1. Logarithms are inverses of exponentials—they answer "what power?"
2. Three important bases: e (calculus), 10 (scientific), 2 (computing)
3. Properties convert operations: multiplication → addition, powers → multiplication
4. Real-world scales (Richter, decibels, pH) compress huge ranges
5. ML relies on logarithms for numerical stability and theoretical elegance
6. Always check for edge cases: log(0), log(negative), precision for small arguments

Exercises

Conceptual Questions

1. Explain in your own words why $log_b(1) = 0$ for any valid base $b$.
2. Why can't we take the logarithm of a negative number using real numbers?
3. If $log_2(n) = 10$, how many elements are in the set? How many comparisons would binary search take?
4. A sound at 60 dB is how many times more intense than a sound at 40 dB?

Computational Problems

1. Simplify: log_3(81) + log_3left(rac{1}{27} ight)
2. Solve for x: $log_2(x-3) + log_2(x+3) = 4$
3. Express $log_5(100)$ in terms of common logarithms (base 10).
4. If an earthquake releases 1000 times more energy than a magnitude 4 earthquake, what is its magnitude? (Hint: energy ratio ≈ $31.6^{\Delta M}$)

Programming Challenges

1. Implement a function that computes $log_b(x)$ for any base using only the natural log.
2. Write a numerically stable function to compute cross-entropy loss given predicted probabilities and true labels.
3. Create a visualization comparing linear and logarithmic scales for the electromagnetic spectrum (wavelengths from 10⁻¹² to 10⁴ meters).

Exploration

1. Research Benford's Law: why do leading digits in many datasets follow a logarithmic distribution?
2. Investigate how logarithms appear in the analysis of algorithm complexity (e.g., why is merge sort O(n log n)?).
3. Explore the connection between logarithms and music: why is the frequency ratio between octaves 2:1, and how do logarithms relate to the perception of pitch?

In the next section, we'll explore trigonometric functions—the mathematics of circular motion and periodic phenomena. These functions are essential for understanding waves, oscillations, rotations, and countless applications in physics, engineering, and signal processing.