Matrix Multiplication in Networks

Learning Objectives

By the end of this section, you will be able to:

1. Rewrite per-neuron calculations as matrix multiplications and verify they produce identical results
2. Read dimension annotations and know the shape of every tensor at each stage
3. Prove why non-linearity is essential — without it, deep networks collapse to shallow ones
4. Implement the forward pass in Python using NumPy in just 3 lines

Why this matters: In Section 1, we computed each neuron one at a time — 12 separate multiplications for each layer. Matrix multiplication does an entire layer in one operation. Every framework (PyTorch, TensorFlow) works this way. The code mirrors the math line for line.

From Neurons to One Matrix Operation

In Section 1, we computed each hidden neuron separately. Look at the pattern:

$z_0 = x_0 W_1[0][0] + x_1 W_1[1][0] + x_2 W_1[2][0] + x_3 W_1[3][0] + b_1[0]$
$z_1 = x_0 W_1[0][1] + x_1 W_1[1][1] + x_2 W_1[2][1] + x_3 W_1[3][1] + b_1[1]$
$z_2 = x_0 W_1[0][2] + x_1 W_1[1][2] + x_2 W_1[2][2] + x_3 W_1[3][2] + b_1[2]$

Each equation is a dot product between $\mathbf{x}$ and a column of $\mathbf{W}_1$. All three at once is a matrix-vector multiplication: $\mathbf{z}^{(1)} = \mathbf{x} \cdot \mathbf{W}_1 + \mathbf{b}_1$

In NumPy this is just: $\texttt{z1 = x @ W1 + b1}$. One line replaces three calculations.

Layer 1 as Matrix Multiplication

$\mathbf{x} \cdot \mathbf{W}_1 = [1, 0, 1, 1] \cdot \begin{bmatrix} 0.2 & -0.5 & 0.1 \\ -0.3 & 0.4 & -0.2 \\ 0.1 & 0.3 & 0.5 \\ -0.4 & 0.2 & -0.1 \end{bmatrix}$

The vector $\mathbf{x}$ dots with each column:

Column 0: $(1)(0.2) + (0)(-0.3) + (1)(0.1) + (1)(-0.4) = -0.1$
Column 1: $(1)(-0.5) + (0)(0.4) + (1)(0.3) + (1)(0.2) = 0.0$
Column 2: $(1)(0.1) + (0)(-0.2) + (1)(0.5) + (1)(-0.1) = 0.5$

$\mathbf{x} \cdot \mathbf{W}_1 + \mathbf{b}_1 = [-0.1, 0.0, 0.5] + [0.1, -0.1, 0.0] = [0.0, -0.1, 0.5]$ — same as Section 1.

Layer 2 as Matrix Multiplication

After ReLU: $\mathbf{h} = [0.0, 0.0, 0.5]$. The output layer is $\hat{\mathbf{y}} = \mathbf{h} \cdot \mathbf{W}_2 + \mathbf{b}_2$.

Since $h_0 = h_1 = 0$, only the third row of $\mathbf{W}_2$ contributes: $[0.5 \times 0.2,\; 0.5 \times (-0.4),\; 0.5 \times 0.1,\; 0.5 \times (-0.5)] = [0.1, -0.2, 0.05, -0.25]$

Adding bias: $[0.1, -0.2, 0.05, -0.25] + [0.0, 0.1, -0.1, 0.0] = [0.1, -0.1, -0.05, -0.25]$ — matches Section 1 exactly.

The Complete Forward Pass Equation

Two lines of math, or one combined:

$\mathbf{h} = \text{ReLU}(\mathbf{x} \cdot \mathbf{W}_1 + \mathbf{b}_1)$
$\hat{\mathbf{y}} = \mathbf{h} \cdot \mathbf{W}_2 + \mathbf{b}_2$

Dimension annotations at each step:

Why Non-Linearity Is Essential

Critical question: what if we remove ReLU and just stack two linear layers?

$\hat{\mathbf{y}} = (\mathbf{x} \cdot \mathbf{W}_1 + \mathbf{b}_1) \cdot \mathbf{W}_2 + \mathbf{b}_2 = \mathbf{x} \cdot \mathbf{W}_1 \mathbf{W}_2 + \mathbf{b}_1 \mathbf{W}_2 + \mathbf{b}_2$

Define $\mathbf{W}_{\text{eff}} = \mathbf{W}_1 \mathbf{W}_2$ and $\mathbf{b}_{\text{eff}} = \mathbf{b}_1 \mathbf{W}_2 + \mathbf{b}_2$. Then: $\hat{\mathbf{y}} = \mathbf{x} \cdot \mathbf{W}_{\text{eff}} + \mathbf{b}_{\text{eff}}$

This is just one linear layer! Two linear layers without activation collapse into a single linear transformation. You could stack 100 layers and it'd still be equivalent to one.

Proof with our numbers

Let's verify with the exact weights from Section 1. Compute the effective single-layer matrix $\mathbf{W}_{\text{eff}} = \mathbf{W}_1 \mathbf{W}_2$ (shape 4×4):

$\mathbf{W}_{\text{eff}} = \begin{bmatrix} 0.2 & -0.5 & 0.1 \\ -0.3 & 0.4 & -0.2 \\ 0.1 & 0.3 & 0.5 \\ -0.4 & 0.2 & -0.1 \end{bmatrix} \begin{bmatrix} 0.3 & -0.2 & 0.4 & 0.1 \\ -0.1 & 0.5 & -0.3 & 0.2 \\ 0.2 & -0.4 & 0.1 & -0.5 \end{bmatrix} = \begin{bmatrix} 0.09 & -0.33 & 0.24 & -0.05 \\ -0.17 & 0.34 & -0.26 & 0.15 \\ 0.10 & -0.17 & 0.00 & -0.18 \\ -0.16 & 0.22 & -0.23 & 0.02 \end{bmatrix}$

And the effective bias: $\mathbf{b}_{\text{eff}} = \mathbf{b}_1 \cdot \mathbf{W}_2 + \mathbf{b}_2 = [0.04,\; 0.03,\; -0.03,\; -0.01]$

Now compute the output using just this single layer: $\mathbf{x} \cdot \mathbf{W}_{\text{eff}} + \mathbf{b}_{\text{eff}} = [1,0,1,1] \cdot \mathbf{W}_{\text{eff}} + \mathbf{b}_{\text{eff}} = [0.07,\; -0.25,\; 0.01,\; -0.22]$ — identical to running the two-layer network without ReLU. Two layers bought us nothing.

Dimensions Cheat Sheet

General pattern for any layer:

Python Implementation

Here's the complete forward pass in Python with NumPy. Click any line to see the exact values flowing through. The core is just 3 lines: $\texttt{z1 = x @ W1 + b1}$, $\texttt{h = np.maximum(0, z1)}$, $\texttt{y\_hat = h @ W2 + b2}$.

       h0    h1    h2
x0   0.2  -0.5   0.1
x1  -0.3   0.4  -0.2
x2   0.1   0.3   0.5
x3  -0.4   0.2  -0.1

       y0    y1    y2    y3
h0   0.3  -0.2   0.4   0.1
h1  -0.1   0.5  -0.3   0.2
h2   0.2  -0.4   0.1  -0.5

1  0
1  1

       h0    h1    h2
x0   0.2  -0.5   0.1
x1  -0.3   0.4  -0.2
x2   0.1   0.3   0.5
x3  -0.4   0.2  -0.1

       y0    y1    y2    y3
h0   0.3  -0.2   0.4   0.1  ← ×0.0 (dead)
h1  -0.1   0.5  -0.3   0.2  ← ×0.0 (dead)
h2   0.2  -0.4   0.1  -0.5  ← ×0.5 (alive!)

1  1
0  1

1import numpy as np
2
3# ── Network weights (same values from Section 1) ──
4W1 = np.array([
5    [ 0.2, -0.5,  0.1],
6    [-0.3,  0.4, -0.2],
7    [ 0.1,  0.3,  0.5],
8    [-0.4,  0.2, -0.1]
9])
10b1 = np.array([0.1, -0.1, 0.0])
11
12W2 = np.array([
13    [ 0.3, -0.2,  0.4,  0.1],
14    [-0.1,  0.5, -0.3,  0.2],
15    [ 0.2, -0.4,  0.1, -0.5]
16])
17b2 = np.array([0.0, 0.1, -0.1, 0.0])
18
19# ── Input image ──
20image = np.array([[1, 0],
21                  [1, 1]])
22x = image.flatten().astype(float)
23
24# ── Forward pass (one matrix op per layer) ──
25z1 = x @ W1 + b1
26h = np.maximum(0, z1)
27y_hat = h @ W2 + b2
28
29# ── Target and loss ──
30target = image.T.flatten().astype(float)
31mse = np.mean((y_hat - target) ** 2)
32
33print(f"Input:      {x}")
34print(f"z1:         {z1}")
35print(f"h (ReLU):   {h}")
36print(f"Prediction: {np.round(y_hat, 4)}")
37print(f"Target:     {target}")
38print(f"MSE Loss:   {mse:.4f}")

Summary

1. Matrix multiplication replaces loops. One $\texttt{x @ W1 + b1}$ computes all 3 hidden neurons at once.
2. Two matrix multiplications + ReLU = complete forward pass. Three lines of Python.
3. Without activation, depth is useless. Multiple linear layers collapse to one.
4. Always check dimensions. (a,) × (a, b) = (b,). Inner must match.

In the next section, we'll implement this in PyTorch using $\texttt{nn.Module}$ — the production way to build neural networks — and create the full training dataset.