Multi-Layer Perceptrons (MLPs)

Learning Objectives

The Story Behind MLPs

The story of Multi-Layer Perceptrons is one of the most dramatic tales in the history of artificial intelligence. It involves a crisis, a decade of doubt, and ultimately a triumphant comeback.

The Rise of the Perceptron (1958)

In 1958, Frank Rosenblatt introduced the Perceptron, a simple computational model inspired by biological neurons. The New York Times declared it "the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence."

The perceptron was remarkably good at learning linearly separable patterns—problems where a straight line (or hyperplane) could separate different classes. It could learn to distinguish AND, OR, and even recognize simple patterns.

The Fall: Minsky and Papert's Critique (1969)

In 1969, Marvin Minsky and Seymour Papert published "Perceptrons", a mathematical analysis proving fundamental limitations of single-layer perceptrons. Most devastatingly, they showed that a single perceptron cannot learn the XOR function.

The XOR Problem: XOR (exclusive or) outputs 1 when exactly one input is 1. No single straight line can separate the (0,0)→0, (1,1)→0 points from the (0,1)→1, (1,0)→1 points. This proved that perceptrons couldn't solve even simple non-linear problems.

This critique, combined with the overhyped promises, triggered the first "AI Winter"—a period of reduced funding and interest in neural network research that lasted over a decade.

The Resurrection: Hidden Layers and Backpropagation (1986)

The solution was actually known but not widely appreciated: add hidden layers. In 1986, Rumelhart, Hinton, and Williams popularized the backpropagation algorithm, demonstrating how to train multi-layer networks efficiently.

The XOR Problem: Why One Layer Isn't Enough

To truly understand why MLPs were revolutionary, we need to understand the XOR problem deeply. Let's look at the truth table:

Notice how the outputs form a checkerboard pattern: opposite corners have the same label. This is the defining characteristic of XOR—and why it's impossible to separate with a single line.

Mathematical Proof

A single perceptron computes:

The decision boundary is where $w_1 x_1 + w_2 x_2 + b = 0$, which is a straight line. No matter how you rotate or shift this line, you cannot separate the XOR points!

Interactive: The XOR Problem

Experiment with different problems (AND, OR, XOR) and network architectures. Notice how XOR cannot be solved by a single-layer perceptron, but can be solved when you add a hidden layer.

MLP Architecture

A Multi-Layer Perceptron consists of layers of neurons, where each neuron in one layer connects to every neuron in the next layer. This is why MLPs are also called fully connected or dense networks.

Components of an MLP

1. Input Layer: Receives the raw features. The number of neurons equals the number of input features. This layer performs no computation—it just passes data forward.
2. Hidden Layers: One or more layers between input and output. Each neuron computes a weighted sum of its inputs, adds a bias, and applies an activation function. These layers learn intermediate representations.
3. Output Layer: Produces the final prediction. The number of neurons depends on the task: 1 for binary classification, C for C-class classification, or any number for regression.

Key Terminology

Interactive: Network Architecture

Explore how changing the number of neurons in each layer affects the network structure. Watch how connections are drawn between layers and how activations flow during forward propagation.

Forward Propagation

Forward propagation is the process of computing the network's output given an input. Data flows from the input layer through hidden layers to the output layer, with each neuron performing the same basic computation.

The Neuron Computation

Each neuron performs three steps:

1. Weighted Sum: Multiply each input by its corresponding weight and sum them:
   $$z = \sum_{i=1}^{n} w_i x_i = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n$$
2. Add Bias: Add a learnable bias term:
   $$z = \sum_{i=1}^{n} w_i x_i + b$$
3. Apply Activation: Pass through a non-linear activation function:
   $$a = \sigma(z)$$

Interactive: Step-by-Step Forward Pass

Step through a complete forward pass and see exactly how inputs are transformed at each stage. Watch the weighted sums and activation values being computed in real-time.

The Magic of Hidden Layers

Hidden layers are where the real magic happens. They transform the input space in a way that makes previously unsolvable problems suddenly solvable. But how?

Feature Learning

Each hidden neuron creates a new feature by computing a weighted combination of inputs. These learned features capture patterns in the data that are useful for the task.

Geometric Interpretation: Each hidden neuron draws a hyperplane through the input space. Points on one side activate the neuron (output near 1), while points on the other side don't (output near 0). The hidden layer output is a new coordinate system based on distances from these hyperplanes!

Solving XOR with Hidden Layers

For XOR, we need two hidden neurons that together create a new feature space where the points become linearly separable:

• Hidden Neuron 1: Fires when $x_1 + x_2 > 0.5$ (at least one input is 1)
• Hidden Neuron 2: Fires when $x_1 + x_2 > 1.5$ (both inputs are 1)

In this new (h₁, h₂) space:

• (0,0) → (0, 0): Both neurons inactive → output 0
• (0,1) or (1,0) → (1, 0): First neuron active → output 1
• (1,1) → (1, 1): Both neurons active → output 0

Now a simple line like $h_1 - h_2 = 0.5$ can separate the classes!

Interactive: Space Transformation

This visualization shows how hidden layers warp the input space. Adjust the weights of two hidden neurons and watch how the XOR points get rearranged in the hidden layer's feature space.

Activation Functions

Activation functions introduce non-linearity into the network. Without them, no matter how many layers we stack, the network can only learn linear transformations.

Common Activation Functions

ReLU: The Modern Default

ReLU (Rectified Linear Unit) has become the default activation for hidden layers because:

• Efficient: Just a comparison operation—much faster than exp() in sigmoid/tanh
• No vanishing gradient: Gradient is 1 for positive inputs, not squashed like sigmoid
• Sparse activation: Negative inputs produce 0, creating sparse representations

Mathematical Formulation

Let's formalize the MLP mathematically. Consider a network with $L$ layers (including input and output).

Notation

• $\mathbf{a}^{(l)}$: Activation vector at layer $l$
• $W^{(l)}$: Weight matrix from layer $l-1$ to $l$
• $\mathbf{b}^{(l)}$: Bias vector at layer $l$
• $\sigma^{(l)}$: Activation function at layer $l$

Forward Pass Equations

Starting with input $\mathbf{a}^{(0)} = \mathbf{x}$, for each layer $l = 1, 2, \ldots, L$:

The final output is $\hat{\mathbf{y}} = \mathbf{a}^{(L)}$.

Dimensionality

For a layer mapping $n_{l-1}$ inputs to $n_l$ outputs:

• $W^{(l)}$ has shape $n_l \times n_{l-1}$
• $\mathbf{b}^{(l)}$ has shape $n_l$
• Total parameters for this layer: $n_l \times n_{l-1} + n_l = n_l(n_{l-1} + 1)$

PyTorch Implementation

Let's implement a flexible MLP in PyTorch that can have any number of hidden layers. Click on code lines to see detailed explanations.

class MyModel(nn.Module):

hidden_dims=[64, 32] creates two hidden layers

super().__init__()

nn.Linear(2, 4) has 2*4=8 weights and 4 biases

Sequential(Linear, ReLU, Linear)

1import torch
2import torch.nn as nn
3import torch.nn.functional as F
4
5class MLP(nn.Module):
6    """Multi-Layer Perceptron for classification"""
7
8    def __init__(self, input_dim, hidden_dims, output_dim):
9        super().__init__()
10
11        # Build layers dynamically
12        layers = []
13        prev_dim = input_dim
14
15        # Hidden layers
16        for hidden_dim in hidden_dims:
17            layers.append(nn.Linear(prev_dim, hidden_dim))
18            layers.append(nn.ReLU())
19            prev_dim = hidden_dim
20
21        # Output layer
22        layers.append(nn.Linear(prev_dim, output_dim))
23
24        self.network = nn.Sequential(*layers)
25
26    def forward(self, x):
27        return self.network(x)
28
29# Create an MLP: 2 inputs -> [4, 3] hidden -> 1 output
30model = MLP(input_dim=2, hidden_dims=[4, 3], output_dim=1)
31
32# Example forward pass
33x = torch.tensor([[0.5, 0.8]])
34output = model(x)
35print(f"Input: {x}")
36print(f"Output: {output}")

Interactive: MLP Playground

Experiment with different datasets, network architectures, and hyperparameters. Watch how the decision boundary evolves during training.

MLP Design Choices

Width vs Depth

Should you make your network wider (more neurons per layer) or deeper (more layers)? The answer depends on the problem:

Universal Approximation Theorem: A single hidden layer with enough neurons can approximate any continuous function. However, deeper networks can represent the same function with exponentially fewer parameters.

Practical Guidelines

1. Start small: Begin with 1-2 hidden layers and increase if needed
2. Power of 2: Layer sizes like 32, 64, 128, 256 are common (GPU efficiency)
3. Tapered: Decreasing layer sizes (e.g., 256→128→64) often work well
4. ReLU for hidden: Use ReLU in hidden layers unless you have a specific reason not to
5. Output activation: Sigmoid for binary classification, softmax for multiclass, none for regression

Common Pitfalls

Test Your Understanding

Summary

In the next section, we'll explore the Universal Approximation Theorem in depth—the mathematical foundation that explains why neural networks are such powerful function approximators.