The Dot Product

Learning Objectives

By the end of this section, you will be able to:

1. Compute the dot product of two vectors using both the algebraic and geometric definitions
2. Interpret the dot product geometrically as a measure of how much two vectors point in the same direction
3. Apply the dot product to find angles between vectors
4. Determine when two vectors are orthogonal (perpendicular) using the dot product
5. Calculate the projection of one vector onto another
6. Use the dot product in physics applications, especially work and force
7. Connect the dot product to cosine similarity in machine learning

The Big Picture: Measuring Alignment

"The dot product answers a fundamental question: How much do two vectors point in the same direction?"

Imagine you're pushing a heavy box across the floor. You push with a certain force, but the box only moves horizontally. How much of your effort actually moves the box? If you push straight down, you do no useful work — your force is perpendicular to the motion. If you push parallel to the floor, all your effort moves the box. For any angle in between, only part of your force contributes.

The dot product (also called the scalar product or inner product) precisely quantifies this relationship. It takes two vectors and produces a single number that measures their "alignment."

Where the Dot Product Appears

Historical Context: The Birth of Vector Algebra

The dot product emerged in the 19th century as mathematicians developed the modern concept of vectors. William Rowan Hamilton (1805–1865) invented quaternions in 1843, which contained both a scalar and vector part. When you "multiply" two quaternions, the scalar part of the result is related to what we now call the dot product.

Josiah Willard Gibbs (1839–1903) and Oliver Heaviside (1850–1925) later extracted the dot and cross products as separate operations, creating the vector algebra we use today. Gibbs developed this specifically for his work in physics, where he needed to express work and energy in terms of forces and displacements.

Algebraic Definition

We begin with the component-by-component definition, which is how we actually compute dot products.

Example: Computing a Dot Product

Let $\mathbf{a} = \langle 3, -2, 5 \rangle$ and $\mathbf{b} = \langle 4, 1, -2 \rangle$. Find $\mathbf{a} \cdot \mathbf{b}$.

Interactive: Exploring the Dot Product

Use this interactive visualizer to develop intuition for the dot product. Drag the vector endpoints or use the sliders to see how the dot product changes with the angle between vectors.

Geometric Interpretation

The algebraic definition tells us how to compute the dot product. The geometric definition tells us what it means.

Understanding the Geometric Formula

The formula $\mathbf{a} \cdot \mathbf{b} = |\mathbf{a}| |\mathbf{b}| \cos\theta$ reveals the true meaning of the dot product:

• The dot product measures alignment. When $\theta = 0°$ (parallel vectors), $\cos\theta = 1$, and the dot product is maximized.
• Perpendicular vectors have zero dot product. When $\theta = 90°$, $\cos\theta = 0$.
• Opposite vectors give negative dot products. When $\theta = 180°$, $\cos\theta = -1$.
• The result scales with both magnitudes. Longer vectors produce larger (in absolute value) dot products.

Properties of the Dot Product

The dot product satisfies several important algebraic properties that make it useful for calculations.

Finding Angles Between Vectors

One of the most important applications of the dot product is finding the angle between two vectors. Rearranging the geometric formula:

Therefore: $\theta = \arccos\left(\frac{\mathbf{a} \cdot \mathbf{b}}{|\mathbf{a}| \, |\mathbf{b}|}\right)$

Example: Finding the Angle

Find the angle between $\mathbf{a} = \langle 2, 3 \rangle$ and $\mathbf{b} = \langle 4, -1 \rangle$.

Orthogonality: Perpendicular Vectors

Two vectors are orthogonal (perpendicular) if and only if their dot product is zero.

This is because when $\theta = 90°$, $\cos 90° = 0$, making the entire product zero.

Example: Testing for Orthogonality

Are $\mathbf{u} = \langle 2, -3, 1 \rangle$ and $\mathbf{v} = \langle 4, 3, 1 \rangle$ orthogonal?

Vector Projection

The projection of $\mathbf{a}$ onto $\mathbf{b}$ answers the question: "How much of $\mathbf{a}$ lies in the direction of $\mathbf{b}$?"

Scalar Projection (Component)

The scalar projection (or component) of $\mathbf{a}$ onto $\mathbf{b}$ is the signed length of the projection:

Vector Projection

The vector projection of $\mathbf{a}$ onto $\mathbf{b}$ is a vector in the direction of $\mathbf{b}$ with the same length as the scalar projection:

Vector Decomposition

Any vector can be decomposed into components parallel and perpendicular to another vector:

where $\text{perp}_\mathbf{b} \mathbf{a} = \mathbf{a} - \text{proj}_\mathbf{b} \mathbf{a}$ is the component of $\mathbf{a}$ perpendicular to $\mathbf{b}$.

Interactive: Vector Projection

Explore how the projection of vector $\mathbf{a}$ onto $\mathbf{b}$ works. The purple vector shows the projection, and the green dashed line shows the perpendicular component.

Physics Applications

Work Done by a Force

When a constant force $\mathbf{F}$ moves an object through a displacement $\mathbf{d}$, the work done is:

This formula explains why:

• Pushing parallel to motion ($\theta = 0°$): Maximum work ($W = Fd$)
• Pushing perpendicular to motion ($\theta = 90°$): No work done ($W = 0$)
• Pushing opposite to motion ($\theta = 180°$): Negative work ($W = -Fd$), which slows the object

Example: Work on an Inclined Plane

A force of 50 N is applied at 30° above the horizontal to push a cart 10 m along a flat surface. How much work is done?

Machine Learning Applications

The dot product is fundamental to modern machine learning, appearing in everything from basic similarity measures to the attention mechanisms that power transformers.

Cosine Similarity

Cosine similarity normalizes the dot product by the vector magnitudes:

This gives a similarity score between -1 (opposite) and 1 (identical direction), regardless of vector magnitudes. This is crucial in NLP where document length shouldn't affect similarity.

Applications in Modern ML

• Word Embeddings: Words like "king" and "queen" have similar vectors (high cosine similarity), while "king" and "car" are dissimilar
• Attention Mechanisms: In transformers, attention scores are computed using dot products between query and key vectors
• Recommendation Systems: Users with similar preference vectors receive similar recommendations
• Information Retrieval: Search results are ranked by cosine similarity between query and document vectors

Python Implementation

Computing Dot Products

For a = (3, 4) and b = (2, -1): 3×2 + 4×(-1) = 6 - 4 = 2

1import numpy as np
2
3def dot_product_algebraic(a, b):
4    """
5    Compute dot product using the algebraic definition:
6    a · b = a₁b₁ + a₂b₂ + ... + aₙbₙ
7
8    This is equivalent to element-wise multiplication
9    followed by summation.
10    """
11    return np.sum(a * b)
12
13def dot_product_geometric(a, b):
14    """
15    Compute dot product using the geometric definition:
16    a · b = |a| |b| cos(θ)
17
18    First find magnitudes, then the angle between them.
19    """
20    mag_a = np.linalg.norm(a)
21    mag_b = np.linalg.norm(b)
22
23    if mag_a == 0 or mag_b == 0:
24        return 0.0
25
26    # Find angle using the algebraic formula
27    cos_theta = np.dot(a, b) / (mag_a * mag_b)
28
29    # Clamp to [-1, 1] to handle numerical errors
30    cos_theta = np.clip(cos_theta, -1, 1)
31
32    return mag_a * mag_b * cos_theta
33
34# Example vectors
35a = np.array([3, 4])
36b = np.array([2, -1])
37
38print("Vector a:", a)
39print("Vector b:", b)
40print()
41
42# Compute dot product both ways
43algebraic = dot_product_algebraic(a, b)
44geometric = dot_product_geometric(a, b)
45
46print(f"Dot product (algebraic): {algebraic}")
47print(f"Dot product (geometric): {geometric:.6f}")
48print()
49
50# Find the angle between vectors
51cos_theta = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
52theta_rad = np.arccos(np.clip(cos_theta, -1, 1))
53theta_deg = np.degrees(theta_rad)
54
55print(f"Angle between vectors: {theta_deg:.2f}°")
56print(f"cos(θ) = {cos_theta:.4f}")
57print()
58
59# Check orthogonality
60orthogonal_example = np.array([3, 0])
61perpendicular = np.array([0, 4])
62print(f"Orthogonal check: {orthogonal_example} · {perpendicular} = {np.dot(orthogonal_example, perpendicular)}")

Vector Projection

Like dropping a shadow of a onto the line containing b

1import numpy as np
2
3def vector_projection(a, b):
4    """
5    Compute the vector projection of a onto b.
6
7    Formula: proj_b(a) = (a · b / |b|²) b
8
9    This gives the component of a in the direction of b.
10    """
11    dot_ab = np.dot(a, b)
12    dot_bb = np.dot(b, b)  # |b|² = b · b
13
14    if dot_bb == 0:
15        return np.zeros_like(a)
16
17    scalar_proj = dot_ab / dot_bb
18    return scalar_proj * b
19
20def scalar_projection(a, b):
21    """
22    Compute the scalar projection (component) of a onto b.
23
24    Formula: comp_b(a) = a · b / |b|
25
26    This is the signed length of the projection.
27    """
28    mag_b = np.linalg.norm(b)
29    if mag_b == 0:
30        return 0.0
31    return np.dot(a, b) / mag_b
32
33def decompose_vector(a, b):
34    """
35    Decompose vector a into parallel and perpendicular
36    components relative to b.
37
38    a = proj_b(a) + perp_b(a)
39    """
40    parallel = vector_projection(a, b)
41    perpendicular = a - parallel
42    return parallel, perpendicular
43
44# Example: Decompose force vector
45force = np.array([10, 6])      # Force applied
46direction = np.array([1, 0])   # Horizontal direction
47
48print("Force vector:", force)
49print("Direction:", direction)
50print()
51
52# Compute projections
53proj = vector_projection(force, direction)
54comp = scalar_projection(force, direction)
55parallel, perp = decompose_vector(force, direction)
56
57print(f"Vector projection: {proj}")
58print(f"Scalar projection: {comp}")
59print()
60print(f"Parallel component: {parallel}")
61print(f"Perpendicular component: {perp}")
62print()
63
64# Verify decomposition
65reconstructed = parallel + perp
66print(f"Verification: {parallel} + {perp} = {reconstructed}")
67print(f"Matches original: {np.allclose(reconstructed, force)}")

Cosine Similarity for NLP

Two unit vectors pointing the same way have similarity 1.0

1import numpy as np
2
3def cosine_similarity(a, b):
4    """
5    Compute cosine similarity between two vectors.
6
7    Range: [-1, 1]
8    - 1: vectors point in same direction
9    - 0: vectors are orthogonal
10    - -1: vectors point in opposite directions
11
12    This is the normalized dot product.
13    """
14    mag_a = np.linalg.norm(a)
15    mag_b = np.linalg.norm(b)
16
17    if mag_a == 0 or mag_b == 0:
18        return 0.0
19
20    return np.dot(a, b) / (mag_a * mag_b)
21
22# Word embeddings example (simplified 2D)
23word_vectors = {
24    'king':    np.array([0.85, 0.70]),
25    'queen':   np.array([0.75, 0.82]),
26    'man':     np.array([0.55, 0.45]),
27    'woman':   np.array([0.48, 0.60]),
28    'prince':  np.array([0.72, 0.58]),
29}
30
31print("Word Similarity Analysis")
32print("=" * 40)
33
34# Compare king to other words
35reference = 'king'
36ref_vec = word_vectors[reference]
37
38similarities = {}
39for word, vec in word_vectors.items():
40    if word != reference:
41        sim = cosine_similarity(ref_vec, vec)
42        similarities[word] = sim
43
44# Sort by similarity
45sorted_words = sorted(similarities.items(),
46                      key=lambda x: x[1],
47                      reverse=True)
48
49print(f"\nWords most similar to '{reference}':")
50for word, sim in sorted_words:
51    print(f"  {word}: {sim:.4f} ({sim*100:.1f}%)")
52
53# The famous word arithmetic
54print("\n" + "=" * 40)
55print("Word Arithmetic: king - man + woman = ?")
56print("=" * 40)
57
58result = word_vectors['king'] - word_vectors['man'] + word_vectors['woman']
59print(f"Result vector: {result}")
60
61# Find closest word to result
62best_match = None
63best_sim = -1
64for word, vec in word_vectors.items():
65    sim = cosine_similarity(result, vec)
66    if sim > best_sim:
67        best_sim = sim
68        best_match = word
69
70print(f"Closest word: '{best_match}' (similarity: {best_sim:.4f})")

Common Pitfalls

Test Your Understanding

Summary

The dot product is one of the most important operations in vector mathematics, connecting algebra, geometry, physics, and modern machine learning.

Key Formulas

Key Takeaways

1. The dot product measures alignment between vectors — how much they point in the same direction
2. Positive dot product: acute angle (similar directions)
3. Zero dot product: right angle (orthogonal/perpendicular)
4. Negative dot product: obtuse angle (opposite directions)
5. Projection decomposes a vector into parallel and perpendicular components relative to another vector
6. In physics, work = force · displacement
7. In ML, cosine similarity uses the normalized dot product to compare text, images, and embeddings

Coming Next: In the next section, we'll explore the Cross Product, which answers a different question: Given two vectors, what vector is perpendicular to both? This is essential for finding normal vectors, computing torque, and understanding rotational physics.