Linear Layers

Learning Objectives

What is a Linear Layer?

The linear layer (also called a fully connected layer or dense layer) is the most fundamental building block in neural networks. It's called "linear" because it performs a linear transformation on its inputs, and "fully connected" because every input is connected to every output.

The Core Idea: A linear layer takes an input vector, multiplies it by a weight matrix to combine and transform the features, then adds a bias vector to shift the result. This simple operation is the foundation upon which all neural network computation is built.

Think of a linear layer as a learnable function that:

1. Combines features — Each output is a weighted sum of all inputs
2. Changes dimensionality — Can expand, compress, or preserve the number of features
3. Shifts the origin — The bias allows the hyperplane to not pass through zero

Historical Context

The linear layer has roots going back to the 1940s with McCulloch and Pitts' mathematical model of neurons. The perceptron, introduced by Rosenblatt in 1958, was essentially a single linear layer with a step function. Today, linear layers form the backbone of everything from simple classifiers to massive language models like GPT.

Mathematical Foundation

A linear layer computes the affine transformation:

Let's break down each component:

The Input Vector x

The input $\mathbf{x} \in \mathbb{R}^{n}$ is a vector of $n$ features:

These could be pixel values, word embeddings, or features from a previous layer.

The Weight Matrix W

The weight matrix $W \in \mathbb{R}^{m \times n}$ contains the learnable parameters that determine how inputs are combined:

Each row $\mathbf{w}_i$ represents the weights for computing output $y_i$. The weight $w_{ij}$ determines how much input $x_j$ contributes to output $y_i$.

The Bias Vector b

The bias $\mathbf{b} \in \mathbb{R}^{m}$ is added after the matrix multiplication:

The bias allows the layer to shift its output, enabling it to fit functions that don't pass through the origin.

The Full Computation

For each output element $y_i$:

This is simply a dot product between the weight row and the input vector, plus a bias term. The matrix form computes all outputs simultaneously.

Geometric Interpretation

Understanding the geometric meaning of linear layers provides deep insight into what neural networks are doing. A linear layer performs three types of transformations:

1. Rotation and Reflection

The weight matrix can rotate the input space. For example, a 2D rotation by angle $\theta$ uses:

If the determinant of $W$ is negative, the transformation includes a reflection, flipping the orientation of space.

2. Scaling and Shearing

The weight matrix can stretch or compress space along different axes. A pure scaling transformation:

Shearing skews the coordinate system while keeping parallel lines parallel:

3. Translation (via Bias)

The bias vector $\mathbf{b}$ translates (shifts) the entire transformed space. Without bias, all linear transformations would keep the origin fixed.

4. Dimensionality Change

Perhaps most importantly, linear layers can change the dimensionality of data:

• Expansion ($m > n$): Project to higher dimensions. Useful for creating richer representations.
• Compression (m < n): Project to lower dimensions. Forces the network to learn compact representations.
• Preservation ($m = n$): Same dimensions in and out. Transforms features without changing their count.

Interactive: Linear Transformation

Explore how the weight matrix and bias vector transform 2D space. Adjust the sliders to see how different weights create rotations, scaling, shearing, and reflections. Watch how the unit vectors (basis vectors) and unit circle transform.

Weights and Biases: What Do They Learn?

The weights and biases are the learnable parameters that get updated during training. But what do they represent intuitively?

Each Row is a Feature Detector

Each row of the weight matrix can be thought of as a template or feature detector. The output $y_i$ measures how much the input $\mathbf{x}$ resembles the template $\mathbf{w}_i$ (via dot product).

For image classification, early-layer weights often learn edge detectors. For text, they might learn word relationships. The network discovers useful features automatically!

Bias as a Threshold

The bias $b_i$ acts as a threshold or activation offset. A negative bias means the weighted sum must be large enough to overcome it before producing positive output.

When to Disable Bias

While bias is essential in many cases, there are specific scenarios where you should disable bias using nn.Linear(in_features, out_features, bias=False). Understanding when to omit bias can simplify your model and prevent redundant parameters.

Before Normalization Layers

The most common reason to disable bias is when the linear layer is immediately followed by a normalization layer like BatchNorm, LayerNorm, or GroupNorm.

Why? Normalization layers include their own learnable bias (often called "beta" or "shift"). Having a bias in the linear layer AND in the normalization layer is redundant—the normalization step will center the activations anyway, making the linear layer's bias meaningless.

1import torch.nn as nn
2
3# ❌ Redundant: Both layers have bias
4class RedundantNetwork(nn.Module):
5    def __init__(self):
6        super().__init__()
7        self.fc = nn.Linear(256, 512)  # Has bias
8        self.bn = nn.BatchNorm1d(512)  # Also has bias (beta)
9
10# ✅ Correct: Disable bias before normalization
11class EfficientNetwork(nn.Module):
12    def __init__(self):
13        super().__init__()
14        self.fc = nn.Linear(256, 512, bias=False)  # No bias
15        self.bn = nn.BatchNorm1d(512)  # BatchNorm provides the shift
16
17    def forward(self, x):
18        x = self.fc(x)   # Linear transform without bias
19        x = self.bn(x)   # Normalizes and adds learnable shift
20        return x

This pattern saves parameters and avoids redundancy. For a layer with 512 outputs, you save 512 parameters by disabling bias.

In Attention Mechanisms

Transformer attention layers often use bias=False for the Query, Key, and Value projections:

1import torch.nn as nn
2
3class MultiHeadAttention(nn.Module):
4    def __init__(self, d_model, num_heads):
5        super().__init__()
6        # Q, K, V projections often omit bias
7        self.W_q = nn.Linear(d_model, d_model, bias=False)
8        self.W_k = nn.Linear(d_model, d_model, bias=False)
9        self.W_v = nn.Linear(d_model, d_model, bias=False)
10
11        # Output projection sometimes includes bias
12        self.W_o = nn.Linear(d_model, d_model)  # bias=True (default)

Why? The attention scores are computed via dot products, and the scaling and softmax operations make the bias in Q and K projections less meaningful. This is a design choice that varies across architectures—some include bias, some don't.

When Layer Output is Centered

If the expected output of your layer should be zero-centered, you might omit bias. Common examples include:

• Residual connections: The linear layer in a residual branch is often initialized to output near-zero values
• Final layers before softmax: Some architectures omit bias in the logits layer, though this is less common
• Embedding projections: When projecting embeddings that are already centered, bias may be unnecessary

Quick Reference

Interactive: Forward Pass Calculator

Step through the forward pass computation to see exactly how inputs, weights, and biases combine to produce outputs. Modify the values and watch the computation update in real-time.

PyTorch Implementation

PyTorch provides nn.Linear as the standard linear layer implementation. Let's explore how to use it and understand its internals.

Basic Usage

Common for MNIST: 28×28=784 pixels → 256 hidden units

1import torch
2import torch.nn as nn
3
4# Create a linear layer: 784 inputs → 256 outputs
5layer = nn.Linear(in_features=784, out_features=256)
6
7# Check the shapes of parameters
8print(f"Weight shape: {layer.weight.shape}")  # (256, 784)
9print(f"Bias shape: {layer.bias.shape}")      # (256,)
10
11# Total parameters
12total_params = layer.weight.numel() + layer.bias.numel()
13print(f"Total parameters: {total_params}")    # 200,960
14
15# Forward pass with a batch of 32 samples
16x = torch.randn(32, 784)  # 32 samples, 784 features each
17y = layer(x)              # 32 samples, 256 features each
18print(f"Output shape: {y.shape}")  # (32, 256)

Building a Network with Linear Layers

Here's how to combine multiple linear layers into a neural network. Note the activation function between layers!

1import torch.nn as nn
2
3# Building a simple classifier with linear layers
4class SimpleClassifier(nn.Module):
5    def __init__(self, input_dim, hidden_dim, num_classes):
6        super().__init__()
7
8        # First linear layer: input → hidden
9        self.fc1 = nn.Linear(input_dim, hidden_dim)
10
11        # Second linear layer: hidden → output
12        self.fc2 = nn.Linear(hidden_dim, num_classes)
13
14        # Activation function (applied between layers)
15        self.relu = nn.ReLU()
16
17    def forward(self, x):
18        # Layer 1: linear + activation
19        x = self.fc1(x)
20        x = self.relu(x)
21
22        # Layer 2: linear (no activation for logits)
23        x = self.fc2(x)
24        return x
25
26# Create classifier: 784 → 256 → 10
27model = SimpleClassifier(784, 256, 10)
28
29# Count parameters
30total = sum(p.numel() for p in model.parameters())
31print(f"Total parameters: {total}")  # 203,530

Parameter Counting

Understanding how many parameters a layer has is crucial for estimating model size, memory requirements, and computational cost.

The Formula

For a linear layer with $n$ inputs and $m$ outputs:

Examples

For a simple MNIST classifier (784→256→128→10), we have over 235,000 parameters! This is why linear layers are sometimes called "dense"—they have many connections.

Memory Considerations

Each parameter is typically a 32-bit float (4 bytes). Memory requirements:

• 235,146 params × 4 bytes = 940 KB for the model weights
• Training requires additional memory for gradients (roughly 2-3× model size)
• Activations (intermediate values) scale with batch size

Weight Initialization

How you initialize weights significantly affects training dynamics. Poor initialization can cause vanishing gradients (weights too small) or exploding gradients (weights too large).

The Goal

We want the variance of activations to remain stable as data flows through the network. If variance shrinks at each layer, gradients will vanish. If it grows, gradients will explode.

Common Strategies

Var(W) = 2/(fan_in + fan_out)

Var(W) = 2/fan_in

1import torch
2import torch.nn as nn
3import torch.nn.init as init
4
5# Create a linear layer
6layer = nn.Linear(512, 256)
7
8# Default initialization: Kaiming uniform
9print("Default (Kaiming):")
10print(f"  Weight std: {layer.weight.std():.4f}")
11
12# Xavier/Glorot initialization (good for tanh/sigmoid)
13init.xavier_uniform_(layer.weight)
14init.zeros_(layer.bias)
15print("\nAfter Xavier:")
16print(f"  Weight std: {layer.weight.std():.4f}")
17
18# Kaiming/He initialization (good for ReLU)
19init.kaiming_uniform_(layer.weight, nonlinearity='relu')
20init.zeros_(layer.bias)
21print("\nAfter Kaiming:")
22print(f"  Weight std: {layer.weight.std():.4f}")
23
24# Custom initialization
25init.normal_(layer.weight, mean=0, std=0.01)
26init.constant_(layer.bias, 0)
27
28# Check initial output scale with sample input
29x = torch.randn(100, 512)
30y = layer(x)
31print(f"\nOutput stats: mean={y.mean():.4f}, std={y.std():.4f}")

Common Use Cases

Linear layers appear throughout deep learning architectures. Here are the most common patterns:

1. Classification Head

The final layer of a classifier that maps features to class logits:

1# Map 512 features to 1000 ImageNet classes
2classifier_head = nn.Linear(512, 1000)
3
4# Output is raw logits (apply softmax for probabilities)
5logits = classifier_head(features)  # Shape: (batch, 1000)
6probs = F.softmax(logits, dim=-1)   # Class probabilities

2. Embedding Projection

Project embeddings to different dimensions:

1# Project word embeddings to a different space
2embedding_dim = 300  # e.g., GloVe embeddings
3hidden_dim = 512
4
5projection = nn.Linear(embedding_dim, hidden_dim)
6projected = projection(word_vectors)  # (batch, seq, 512)

3. Feature Mixing

Combine features before attention or other operations:

1# Transformer MLP block
2class FeedForward(nn.Module):
3    def __init__(self, d_model, d_ff):
4        super().__init__()
5        self.linear1 = nn.Linear(d_model, d_ff)    # Expand
6        self.linear2 = nn.Linear(d_ff, d_model)    # Project back
7        self.activation = nn.GELU()
8
9    def forward(self, x):
10        return self.linear2(self.activation(self.linear1(x)))

4. Dimensionality Reduction

Compress features to a bottleneck:

1# Autoencoder bottleneck
2encoder = nn.Sequential(
3    nn.Linear(784, 256),
4    nn.ReLU(),
5    nn.Linear(256, 64),   # Compress to 64 dimensions
6    nn.ReLU(),
7    nn.Linear(64, 16)     # Bottleneck: only 16 features!
8)

Common Pitfalls

1# ERROR: Dimension mismatch
2x = torch.randn(32, 3, 28, 28)  # Batch of images
3layer = nn.Linear(784, 256)
4# y = layer(x)  # RuntimeError! Expected (32, 784), got (32, 3, 28, 28)
5
6# FIX: Flatten first
7x_flat = x.flatten(1)  # Shape: (32, 2352) - wait, that's 3*28*28!
8# Need to handle channels correctly or use a flattening layer

Test Your Understanding

Test your knowledge of linear layers with this quiz. Each question has a detailed explanation to reinforce the concepts.

Summary

Exercises

Conceptual Questions

1. Explain geometrically what happens when the weight matrix is the identity matrix and the bias is non-zero.
2. Why can't we initialize all weights to zero? What would happen during training?
3. How does the determinant of the weight matrix relate to whether information is lost or preserved?

Coding Exercises

1. Manual Linear Layer: Implement a linear layer without using nn.Linear. Use raw tensor operations: y = x @ W.T + b. Verify your implementation matches nn.Linear's output.
2. Parameter Count Verification: Create a network with layers [100→50→25→10]. Calculate the parameter count manually, then verify with PyTorch.
3. Initialization Experiment: Train a small network on MNIST with: (a) zero initialization, (b) very large random initialization (std=10), (c) Xavier initialization. Compare training curves.
4. Weight Visualization: Train a linear classifier on MNIST (no hidden layers, just 784→10). Visualize the 10 weight vectors as 28×28 images. What do they show?

Challenge: Low-Rank Linear Layer

A standard linear layer from 1000→1000 has 1,001,000 parameters. Implement a low-rank version: 1000→64→1000 (two linear layers with a 64-dim bottleneck). Compare:

• How many parameters does each version have?
• What trade-off are you making?
• When might low-rank be beneficial?

In the next section, we'll explore Activation Functions—the non-linear transformations that give neural networks their power to learn complex patterns. Without them, all those linear layers would collapse into a single transformation!