Vanilla RNN Implementation

NumPy provides the ndarray type and all the linear algebra operations we need: matrix multiplication (@), element-wise operations (tanh, exp), and array creation (zeros). Every gradient computation in BPTT is a matrix operation.

We define a tiny vocabulary of 8 words with hand-crafted 2D embeddings, and 4 training sentences labeled as positive (1) or negative (0). This is the same setup from Section 1, now extended with negative words ('hate', 'bad') so the model has something to learn.

A dictionary mapping each word to its 2D embedding vector. In a real system, these would be learned (Word2Vec, GloVe) or come from a pretrained model. Here we hand-craft them: positive words ('love', 'great') have high second components, negative words ('hate', 'bad') have negative second components.

Four training examples: two positive (target=1) and two negative (target=0). Each example is a tuple of (word_list, label). The model will learn to output probability near 1.0 for positive sentences and near 0.0 for negative ones.

We use the exact same weight matrices as Section 1. These are the initial (untrained) weights — BPTT will compute gradients to update them. After training, these weights will have changed to correctly classify all 4 sentences.

Tuple unpacking: sets input_size=2 (each word is a 2D vector) and hidden_size=3 (the hidden state has 3 elements). These dimensions determine the shapes of all weight matrices.

Transforms each 2D input embedding into the 3D hidden space. Same initial values as Section 1. These will be updated by gradient descent during training.

      x0    x1
h0 [ 0.5  -0.3]
h1 [ 0.2   0.7]
h2 [-0.1   0.4]

The recurrence matrix — transforms the previous hidden state to influence the current one. This is the heart of the RNN's memory. BPTT computes gradients that flow backward through this matrix at every timestep.

       h0    h1    h2
h0 [ 0.1   0.4  -0.2]
h1 [-0.3   0.2   0.5]
h2 [ 0.3  -0.1   0.1]

Hidden bias initialized to zeros. Will be learned during training.

Projects the final hidden state into a single sentiment score. Shape (1, 3) because we have 1 output and 3 hidden units.

Output bias. Adds a small constant to the logit before sigmoid.

This forward pass is almost identical to Section 1, but with one critical difference: it stores ALL intermediate hidden states in h_list. BPTT needs these stored states to compute gradients — you cannot compute ∂L/∂W_hh at timestep t without knowing h_{t-1} and h_t.

Runs the RNN forward pass through the entire sequence, storing every hidden state for later use in BPTT. Returns the list of hidden states, raw logit, and sigmoid probability.

The embedding matrix for one sentence. Each row is a word's 2D embedding.
Example for 'I love this movie':
[[1.0, 0.0],
 [0.5, 0.8],
 [0.3, 0.7],
 [0.9, 0.1]]

Gets the number of timesteps (words) in this sentence. X has shape (T, input_size), so X.shape[0] = T.

Initialize the list with h₀ = [0, 0, 0]. Unlike Section 1 where we overwrote h at each step, here we APPEND each new hidden state. After the loop, h_list = [h₀, h₁, h₂, h₃, h₄] — all 5 states stored for BPTT.

Process each word in sequence. At each step: (1) compute pre-activation from current input and previous hidden state, (2) apply tanh, (3) store the new hidden state. This is identical to Section 1 but now we keep every h_t.

The core RNN equation (pre-activation). Combines current input X[t] and previous hidden state h_list[-1] through two separate learned transformations, then adds bias. This is computed BEFORE tanh.

Apply tanh to squash the pre-activation into (-1, +1), then APPEND to h_list (don't overwrite). This is the critical difference from Section 1 — we preserve every hidden state for BPTT.

Project the final hidden state into a single output value. h_list[-1] is h₄ (after processing all 4 words). W_hy(1,3) @ h₄(3,) gives a scalar logit.

Sigmoid converts the raw logit into a probability between 0 and 1. σ(0.2318) = 0.5577 — barely above 0.5, meaning the untrained model is nearly guessing.

Returns all three values needed for BPTT: h_list for gradient computation, y for debugging, and p for loss calculation.

This is the core of training: Backpropagation Through Time. We compute gradients of the loss with respect to every weight by propagating the error signal backward through the unrolled computation graph — from the output, through h₄, h₃, h₂, h₁, accumulating weight gradients at each step.

Computes all weight gradients using BPTT. Takes the stored forward-pass states and propagates the loss gradient backward through every timestep. Returns gradients for all 5 parameter groups.

Number of timesteps. T=4 for our 4-word sentences.

Initialize gradient accumulators to zero. At each backward timestep, we ADD the local gradient contribution. The final dW_xh is the SUM of contributions from all 4 timesteps — this is because W_xh is shared across time.

Gradient accumulator for the recurrence matrix. Same logic: W_hh is shared across all timesteps, so the total gradient is the sum of per-timestep contributions.

Gradient accumulator for the hidden bias. db_h = Σ_t delta_t.

First, compute the gradient at the output layer. This is the starting point — the error signal that will be propagated backward through time. The output gradient comes from differentiating the binary cross-entropy loss combined with the sigmoid activation.

This is a beautiful simplification: when you combine binary cross-entropy loss L = -[t·log(p) + (1-t)·log(1-p)] with sigmoid p = σ(y), the derivative simplifies to just p - target. No need to differentiate sigmoid and loss separately.

Gradient of loss w.r.t. the output weight matrix. Since y = W_hy @ h_T + b_y, the gradient ∂L/∂W_hy = (∂L/∂y) × h_Tᵀ. This is an outer product: column vector × row vector → matrix.

[[-0.4423]] @ [[0.5597, 0.4605, -0.0660]] = [[-0.2476, -0.2037, 0.0292]]

Gradient of loss w.r.t. output bias. Since y = W_hy @ h + b_y, the derivative ∂L/∂b_y = ∂L/∂y = dL_dy. We copy to avoid aliasing.

Now we need to push the error signal from the output layer INTO the final hidden state h₄. This is the gateway from the output gradients into the BPTT loop.

Compute ∂L/∂h₄ using the chain rule: since y = W_hy @ h₄ + b_y, the gradient flowing backward through this multiplication is W_hyᵀ × dL_dy. This gives us a 3D vector — the error signal for each hidden unit.

This is THE core loop of BPTT. We iterate backward from t=T-1 to t=0 (i.e., from 'movie' back to 'I'), at each step: (1) apply tanh derivative, (2) compute local weight gradients, (3) propagate gradient to the previous timestep via W_hh.

Iterate backward through time: t = 3, 2, 1, 0. At each step, we use the gradient arriving at h_{t+1} (stored in dL_dh) to compute weight gradients and then propagate to h_t. The gradient weakens at each step because it passes through tanh derivatives and W_hh multiplication.

The derivative of tanh(z) is 1 - tanh²(z) = 1 - h_t². Since h_t = tanh(pre_t), we can compute the derivative directly from the stored hidden state — no need to store the pre-activation values.

Element-wise multiply the incoming gradient by the tanh derivative. This is the chain rule at the tanh operation: ∂L/∂(pre_t) = ∂L/∂h_t × ∂h_t/∂(pre_t). The tanh derivative acts as a gate — it can shrink or pass the gradient for each hidden unit independently.

Accumulate the gradient for W_xh at this timestep. Since h_t = tanh(W_xh @ x_t + ...), the gradient ∂L/∂W_xh at time t is delta_t × x_tᵀ (outer product). We ADD to the accumulator because W_xh is shared across all timesteps.

         x0       x1
h0 [-0.1640  -0.0182]
h1 [ 0.1255   0.0139]
h2 [-0.1189  -0.0132]

Accumulate the gradient for W_hh at this timestep. Since pre_t = ... + W_hh @ h_{t-1} + ..., the gradient ∂L/∂W_hh at time t is delta_t × h_{t-1}ᵀ. Note: h_list[t] is h_{t-1} because h_list[0]=h₀.

         h0       h1       h2
h0 [-0.0130  -0.1188  -0.0506]
h1 [ 0.0099   0.0909   0.0387]
h2 [-0.0094  -0.0861  -0.0367]

Accumulate bias gradient. Since pre_t = ... + b_h, the gradient ∂L/∂b_h = delta_t (the bias derivative is always 1). Sum across timesteps.

THIS IS THE KEY STEP OF BPTT. The gradient flows from h_t to h_{t-1} through the recurrence matrix W_hh. Since pre_t depends on h_{t-1} via W_hh @ h_{t-1}, the backward gradient is W_hhᵀ @ delta_t. This multiplication happens at EVERY backward step — it is the second source of gradient decay (or explosion).

     h0     h1    h2
h0 [ 0.1  -0.3   0.3]
h1 [ 0.4   0.2  -0.1]
h2 [-0.2   0.5   0.1]

Return all 5 gradient tensors. These will be used by the optimizer (SGD) to update the weights. Each gradient has the same shape as its corresponding weight.

[[-0.1683, -0.0710],
 [ 0.0984, -0.0312],
 [-0.1073,  0.0602]]

[[-0.0186, -0.1614, -0.0886],
 [-0.0116,  0.0743,  0.0361],
 [ 0.0115, -0.0425, -0.0073]]

[[-0.2476, -0.2037, 0.0292]]

Now we put forward and backward together in a training loop. For each epoch: (1) forward pass on all 4 sentences, (2) BPTT backward on each, (3) average gradients, (4) clip gradients for stability, (5) SGD weight update.

The step size for gradient descent. Each weight update: w ← w − lr × gradient. With lr=0.5, each step changes weights by half the gradient magnitude. Too large → divergence, too small → slow training.

Train for 30 epochs. Each epoch processes all 4 sentences and updates weights once. By epoch 6, the model achieves 100% accuracy. The remaining epochs increase confidence (pushing predictions further from 0.5).

Track total loss and number of correct predictions for this epoch. Reset to zero at the start of each epoch.

Create a list of 5 zero arrays, one for each weight. We accumulate gradients from all 4 sentences, then average them before updating. This is batch gradient descent (as opposed to updating after each sentence).

Process each sentence: forward pass, compute loss, backward pass (BPTT), accumulate gradients. The weight update happens AFTER all sentences are processed.

Convert word list to embedding matrix by looking up each word in the vocabulary dictionary.

Run the forward pass to get hidden states (for BPTT), raw logit, and probability. All intermediate states are stored in h_list.

Binary cross-entropy loss: L = -[t·log(p) + (1-t)·log(1-p)]. Measures how far the prediction p is from the target. When target=1 and p=0.558: L = -log(0.558) = 0.584. When target=0 and p=0.528: L = -log(1-0.528) = 0.749.

Accumulate loss across all sentences for reporting.

Count correct predictions. A prediction is correct if p >= 0.5 matches target=1, or p < 0.5 matches target=0.

Run BPTT to compute all weight gradients for this sentence. Returns a tuple of 5 gradient arrays.

Add this sentence's gradients to the accumulators. After processing all 4 sentences, grads[i] contains the SUM of gradients from all examples.

Average the accumulated gradients by dividing by the number of sentences (4). This gives us the gradient of the AVERAGE loss, which is less noisy than individual sentence gradients.

Gradient clipping is a practical necessity for RNN training. Without it, occasional large gradients can cause weight updates that destroy learned features. We compute the global L2 norm of all gradients and scale them down if the norm exceeds a threshold.

Compute the global L2 norm of ALL gradients concatenated into one vector. This single number tells us how 'large' the total gradient is — if it exceeds our threshold, we scale everything down proportionally.

Only clip if the gradient norm exceeds 5.0. For our small model, this rarely triggers, but for larger models with longer sequences, gradient explosion is common and clipping is essential.

Scale ALL gradients by the same factor so the total norm equals exactly 5.0. This preserves the direction of the gradient (which features to update more/less) while limiting the step size.

Stochastic Gradient Descent: update each weight by subtracting the learning rate times its gradient. The minus sign is because we want to MINIMIZE the loss: move in the opposite direction of the gradient.

Update the recurrence matrix. This changes how the RNN transforms its memory from one timestep to the next.

Update the hidden bias. Shifts the pre-activation values.

Update the output weight matrix. Changes how the final hidden state maps to the prediction.

Update the output bias.

Print progress every 5 epochs. Shows average loss and accuracy improving over time.

Formatted string output showing epoch number, average loss, and number of correct predictions.

PyTorch core library. Provides Tensor objects (like NumPy arrays but with automatic differentiation and GPU support). All neural network computations run on tensors.

Neural network module with building blocks: nn.RNN, nn.Linear, nn.Module, and loss functions like nn.BCEWithLogitsLoss.

Same model architecture as Section 1. nn.Module base class provides automatic parameter tracking, gradient computation, and serialization.

Constructor defining the architecture. Called once to create the model.

Initialize nn.Module internals. Mandatory before defining any layers.

Store for creating h0 in forward(). Just a config value, not a learnable parameter.

Creates the RNN layer. This one line replaces our W_xh, W_hh, b_h AND the entire recurrence loop. PyTorch handles weight initialization, the for loop, and — critically — the gradient computation (BPTT) automatically via autograd.

Output layer: maps 3D hidden state to 1D logit. Replaces our W_hy and b_y.

Forward pass — identical to Section 1. PyTorch records all operations for automatic backpropagation.

Initial hidden state of zeros. Shape: (num_layers=1, batch_size, hidden_size).

Run the entire RNN forward pass. PyTorch internally loops through all timesteps, applying the recurrence at each step. Returns hidden states at all timesteps (output) and the final hidden state (h_n).

Project final hidden states to sentiment logits. squeeze(0) removes the num_layers dimension.

Return raw logits. Sigmoid is applied externally — BCEWithLogitsLoss combines sigmoid + loss for numerical stability.

Create model, loss function, and optimizer. These three objects are the standard PyTorch training setup.

Instantiate the model. Weights are randomly initialized by PyTorch (using Kaiming uniform for RNN and Linear layers). Total: 25 learnable parameters.

Binary Cross-Entropy loss WITH sigmoid built in. This is numerically more stable than applying sigmoid separately and then computing BCE, because it uses the log-sum-exp trick internally.

Stochastic Gradient Descent optimizer. model.parameters() returns ALL learnable parameters (from nn.RNN and nn.Linear). The optimizer stores a reference to these parameters and updates them when optimizer.step() is called.

Unlike NumPy where we processed one sentence at a time, PyTorch processes ALL 4 sentences in a single batch. This is efficient and leverages parallelism.

Creates a single tensor with all training data. Shape (4, 4, 2): 4 sentences × 4 words × 2 embedding dims. Each sentence's embeddings are in a separate 'row' of the batch dimension.

Target labels. Shape (4, 1) to match the model output shape (4, 1).

The PyTorch training loop is remarkably concise. What took ~30 lines of NumPy (forward, backward, gradient accumulation, clipping, update) becomes 6 lines. The key magic: loss.backward() performs BPTT automatically.

Train for 30 epochs, same as NumPy. Each epoch: zero gradients → forward → loss → backward (BPTT) → clip → update.

Clear all parameter gradients from the previous iteration. PyTorch ACCUMULATES gradients by default (for techniques like gradient accumulation), so we must explicitly zero them at the start of each step.

Process ALL 4 sentences in parallel. model(X_train) calls forward(), which runs the RNN on the (4, 4, 2) input tensor. Returns 4 logits in one shot.

Compute the average BCE loss across all 4 sentences. BCEWithLogitsLoss applies sigmoid internally and averages across the batch by default.

THIS SINGLE LINE replaces our entire backward() function. PyTorch's autograd engine traverses the computation graph in reverse, computing gradients for EVERY parameter. For the RNN, this means unrolling through all 4 timesteps — full BPTT — automatically.

Gradient clipping — same concept as our NumPy version. Compute the global L2 norm of all parameter gradients; if it exceeds 5.0, scale all gradients down proportionally.

Apply the SGD update rule to ALL parameters: w ← w − lr × w.grad. This single line replaces our 5 manual update lines (W_xh -= ..., W_hh -= ..., etc.).

Disable gradient tracking for the accuracy computation. We don't need to backpropagate through this — it's just for reporting.

Convert logits to binary predictions: apply sigmoid, threshold at 0.5, convert boolean to float (1.0 or 0.0).

Compute accuracy: compare predictions to targets, convert to float, take the mean.

Print progress every 5 epochs, same as NumPy version.

Formatted progress output showing training convergence.