The Artificial Neuron

From Biology to Mathematics

In the previous section, we saw how biological neurons receive signals through dendrites, process them in the cell body, and fire an output through the axon. The artificial neuron is a mathematical abstraction of this process, distilled to its computational essence.

A biological neuron does three things: (1) it receives many input signals of varying strengths, (2) it combines them \u2014 some excitatory, some inhibitory, (3) if the combined signal exceeds a threshold, it fires. The artificial neuron mirrors this exactly:

The key insight is that we can express the entire neuron as a single equation. Given $n$ inputs, the neuron computes:

$y = f\left(\sum_{i=1}^{n} w_i x_i + b\right) = f(\mathbf{w} \cdot \mathbf{x} + b)$

where $\mathbf{x} = [x_1, x_2, \ldots, x_n]$ is the input vector, $\mathbf{w} = [w_1, w_2, \ldots, w_n]$ is the weight vector, $b$ is the bias, and $f$ is the activation function. Let us understand each piece.

The Weighted Sum

The weighted sum is the neuron's core operation. It takes each input, scales it by a corresponding weight, and adds them all together. Mathematically, this is a dot product:

$z = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b = \mathbf{w} \cdot \mathbf{x} + b$

Think of it this way: each weight $w_i$ tells the neuron how much to care about input $x_i$:

• A large positive weight (e.g., $w_i = 2.0$) means this input is very important and excitatory \u2014 it pushes the neuron toward firing.
• A negative weight (e.g., $w_i = -1.5$) means this input is inhibitory \u2014 it suppresses the neuron's output.
• A weight near zero means the neuron essentially ignores that input.

Intuition: Imagine you're deciding whether to go outside. Temperature (weight: +0.8, you love warmth), rain probability (weight: -1.2, you hate rain), and wind speed (weight: -0.3, mildly annoying). Your brain computes a weighted sum of these factors to make the decision. The artificial neuron does the same thing, but with numbers.

The dot product $\mathbf{w} \cdot \mathbf{x}$ also has a beautiful geometric interpretation: it measures the alignment between the weight vector and the input vector. When the input points in the same direction as the weights, the dot product is large and positive. When they point in opposite directions, it is negative. This is how a neuron learns to detect specific patterns in data.

The Bias Term

The bias $b$ is an additional learnable parameter that shifts the neuron's activation threshold. Without bias, a neuron with all-zero inputs would always produce $z = 0$. The bias allows the neuron to have a non-zero output even when the input is zero.

Geometrically, the bias shifts the decision boundary away from the origin. Consider a 2D neuron with decision boundary $w_1 x_1 + w_2 x_2 + b = 0$. Without bias ($b = 0$), this line must pass through the origin. With bias, the line can be placed anywhere in the plane.

Analogy: The bias is like the y-intercept of a line. In the equation $y = mx + c$, the slope $m$ (weight) controls the angle, but $c$ (bias) controls where the line crosses the y-axis. Without $c$, every line would be forced through the origin.

In practice, $b > 0$ makes the neuron easier to activate (it fires even with weak inputs), while $b < 0$ makes it harder to activate (requires stronger inputs).

Activation Functions

The activation function $f$ is what makes neural networks nonlinear. Without it, a neuron is just a linear function: $y = \mathbf{w} \cdot \mathbf{x} + b$. And a network of linear functions is still linear \u2014 no matter how many layers you stack, the entire network collapses into a single linear transformation.

The activation function is the source of a neural network's power. It allows the network to learn curved decision boundaries, complex patterns, and nonlinear relationships in data. Here are the four most important activation functions:

Step Function (Heaviside)

The original activation from McCulloch-Pitts. Output is binary: $f(z) = 1$ if $z \geq 0$, else $f(z) = 0$. Simple but not differentiable \u2014 gradient descent cannot be applied. Historical importance only.

Sigmoid

The smooth version of the step function: $\sigma(z) = \frac{1}{1 + e^{-z}}$. Maps any real number to the range $(0, 1)$, making it interpretable as a probability. Its derivative is $\sigma'(z) = \sigma(z)(1 - \sigma(z))$, which is always between 0 and 0.25. This means gradients shrink as they propagate backward through many layers \u2014 the vanishing gradient problem.

Tanh (Hyperbolic Tangent)

Similar to sigmoid but zero-centered: $\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$. Output range is $(-1, 1)$. Zero-centered outputs help optimization because the gradient can flow in both positive and negative directions. Mathematically, $\tanh(z) = 2\sigma(2z) - 1$ \u2014 it's a rescaled sigmoid.

ReLU (Rectified Linear Unit)

The modern default: $f(z) = \max(0, z)$. Dead simple, computationally cheap, and avoids vanishing gradients for positive inputs (gradient = 1 when $z > 0$). The only issue: neurons can "die" \u2014 if a neuron's output is always negative, its gradient is permanently zero and it stops learning. Variants like Leaky ReLU ($f(z) = \max(0.01z, z)$) fix this.

The Complete Artificial Neuron

Putting it all together, the artificial neuron computes:

$y = f\left(\sum_{i=1}^{n} w_i x_i + b\right)$

Or in vector form: $y = f(\mathbf{w}^T \mathbf{x} + b)$

The forward pass has exactly two steps:

1. Linear transformation: Compute the pre-activation $z = \mathbf{w} \cdot \mathbf{x} + b$. This is a dot product (measuring alignment between the input and the learned weight pattern) plus a shift.
2. Nonlinear activation: Apply $y = f(z)$. This introduces nonlinearity, enabling the network to learn complex functions.

The learnable parameters are $\mathbf{w}$ (n parameters) and $b$ (1 parameter), for a total of $n + 1$ parameters per neuron. During training, gradient descent adjusts these parameters to minimize the loss function.

Key Insight: A single neuron is a linear classifier. It can separate data that is linearly separable (dividable by a straight line/plane). The activation function determines how confidently it makes this classification. The power of neural networks comes from stacking many neurons \u2014 each subsequent layer can combine the linear boundaries of the previous layer into increasingly complex shapes.

A Single Neuron in Pure Python

Let us implement a single neuron from scratch in Python using NumPy. We will trace the computation step by step so you can see exactly what happens at every line. Click any line in the code to see the detailed execution trace on the left.

1import numpy as np
2
3def neuron(inputs, weights, bias, activation="sigmoid"):
4    """A single artificial neuron: weighted sum + bias + activation."""
5    # Step 1: Compute the weighted sum (dot product)
6    z = np.dot(inputs, weights) + bias
7
8    # Step 2: Apply the activation function
9    if activation == "sigmoid":
10        y = 1 / (1 + np.exp(-z))
11    elif activation == "relu":
12        y = np.maximum(0, z)
13    elif activation == "tanh":
14        y = np.tanh(z)
15    elif activation == "step":
16        y = 1.0 if z >= 0 else 0.0
17    else:
18        y = z  # linear (no activation)
19
20    return y, z
21
22# Example: A neuron with 3 inputs
23inputs = np.array([0.6, 0.4, -0.2])
24weights = np.array([0.7, -0.3, 0.5])
25bias = -0.1
26
27# Forward pass with sigmoid
28output, pre_activation = neuron(inputs, weights, bias, "sigmoid")
29
30print(f"Inputs:   {inputs}")
31print(f"Weights:  {weights}")
32print(f"Bias:     {bias}")
33print(f"")
34print(f"Step 1: z = x . w + b")
35print(f"  = ({inputs[0]}*{weights[0]}) + ({inputs[1]}*{weights[1]}) + ({inputs[2]}*{weights[2]}) + {bias}")
36print(f"  = {inputs[0]*weights[0]:.3f} + {inputs[1]*weights[1]:.3f} + {inputs[2]*weights[2]:.3f} + {bias}")
37print(f"  = {pre_activation:.4f}")
38print(f"")
39print(f"Step 2: y = sigmoid(z) = 1/(1+e^(-{pre_activation:.4f}))")
40print(f"  = {output:.4f}")
41print(f"")
42print(f"Interpretation: The neuron is {output*100:.1f}% confident.")

The output tells us: with these specific weights and inputs, the neuron outputs 0.5250 under sigmoid \u2014 it's barely above 50% confidence. This makes sense: the pre-activation $z = 0.1$ is very close to zero, which is the decision boundary for sigmoid. A slightly more positive z would give higher confidence; a slightly more negative z would flip the prediction.

A Single Neuron in PyTorch

Now let us implement the same neuron in PyTorch. We'll show two approaches: (1) manual computation using torch functions (to prove it's the same math), and (2) using nn.Linear \u2014 the standard PyTorch way that handles weight management, gradient tracking, and GPU acceleration automatically.

Why PyTorch? NumPy computes the forward pass perfectly, but it cannot compute gradients automatically. PyTorch's autograd system records every operation on tensors and can automatically compute $\frac{\partial y}{\partial w_i}$ for all weights simultaneously. This is what makes learning possible.

1import torch
2import torch.nn as nn
3
4# The same neuron, now in PyTorch
5inputs = torch.tensor([0.6, 0.4, -0.2])
6weights = torch.tensor([0.7, -0.3, 0.5])
7bias = torch.tensor([-0.1])
8
9# Method 1: Manual computation (same math as NumPy)
10z = torch.dot(inputs, weights) + bias
11y_sigmoid = torch.sigmoid(z)
12y_relu = torch.relu(z)
13y_tanh = torch.tanh(z)
14
15print("=== Manual Forward Pass ===")
16print(f"z = torch.dot({inputs.tolist()}, {weights.tolist()}) + {bias.item()}")
17print(f"z = {z.item():.4f}")
18print(f"sigmoid(z) = {y_sigmoid.item():.4f}")
19print(f"relu(z)    = {y_relu.item():.4f}")
20print(f"tanh(z)    = {y_tanh.item():.4f}")
21
22# Method 2: Using nn.Linear (the PyTorch way)
23# nn.Linear(in_features=3, out_features=1) creates a neuron
24neuron = nn.Linear(in_features=3, out_features=1, bias=True)
25
26# Manually set weights to match our example
27with torch.no_grad():
28    neuron.weight.copy_(weights.unsqueeze(0))
29    neuron.bias.copy_(bias)
30
31# Forward pass through the nn.Linear layer
32x = inputs.unsqueeze(0)  # Add batch dimension: (3,) -> (1, 3)
33z_linear = neuron(x)
34y_out = torch.sigmoid(z_linear)
35
36print(f"\n=== nn.Linear Forward Pass ===")
37print(f"neuron.weight = {neuron.weight.data}")
38print(f"neuron.bias   = {neuron.bias.data}")
39print(f"z = x @ W.T + b = {z_linear.item():.4f}")
40print(f"sigmoid(z) = {y_out.item():.4f}")

Both methods produce identical results: $z = 0.1000$ and $\sigma(z) = 0.5250$. The nn.Linear version is what real PyTorch networks use \u2014 it handles weight initialization, gradient computation, and device management (CPU/GPU) automatically. When you see nn.Linear(784, 128) in a network, that's 128 neurons, each with 784 inputs.

Geometric Interpretation: Decision Boundaries

A single neuron with inputs $x_1, x_2$, weights $w_1, w_2$, and bias $b$ divides the 2D input space into two regions with the equation:

$w_1 x_1 + w_2 x_2 + b = 0$

This is a straight line (in 2D) or a hyperplane (in higher dimensions). Points on one side satisfy $w_1 x_1 + w_2 x_2 + b > 0$ and are classified as class 1. Points on the other side satisfy $w_1 x_1 + w_2 x_2 + b < 0$ and are classified as class 0.

The weight vector $\mathbf{w} = [w_1, w_2]$ is perpendicular to the decision boundary. It points toward the positive (class 1) region. The bias shifts the boundary away from the origin: positive bias moves it in the negative direction of $\mathbf{w}$, making more of the space classified as positive.

Try adjusting the weights and bias in the visualizer above. Notice how:

• Changing $w_1$ and $w_2$ rotates the decision boundary (changes its angle).
• Changing the bias $b$ translates the boundary (shifts it parallel to itself).
• The green arrow (weight vector) is always perpendicular to the boundary and points toward the blue (class 1) region.
• Misclassified points (yellow rings) cannot be fixed without a nonlinear boundary \u2014 this is the fundamental limitation of a single neuron.

The Neuron as a Binary Classifier

A single neuron with sigmoid activation is a binary classifier: it outputs a probability $p \in (0, 1)$ that the input belongs to class 1. This is precisely logistic regression \u2014 one of the foundational algorithms in machine learning.

The connection is exact:

• Logistic regression: $P(y=1 | \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x} + b)$
• Single neuron: $\text{output} = \sigma(\mathbf{w}^T \mathbf{x} + b)$

They are the same model. A single neuron is logistic regression. The neural network perspective becomes more powerful when we stack multiple neurons into layers, but the fundamental unit is this simple: a dot product, a bias, and a nonlinear squashing function.

What can a single neuron learn? Anything that is linearly separable:

The XOR limitation drove the development of multi-layer networks. By stacking neurons into hidden layers, each layer creates a new linear boundary, and the combination of these boundaries can represent any continuous function \u2014 this is the universal approximation theorem, which we will prove later.

Looking Ahead

You now understand the complete anatomy of a single artificial neuron: inputs multiplied by learned weights, summed with a bias, and transformed by a nonlinear activation function. This is the atom of deep learning.

In the next section, we will survey the different types of neural networks built from these atoms \u2014 feedforward networks, convolutional networks, recurrent networks, and transformers \u2014 each designed for a different type of data and task. But every one of them is built from this same fundamental unit: the artificial neuron.

Remember: A single neuron computes $y = f(\mathbf{w} \cdot \mathbf{x} + b)$. The weights decide what patterns to look for. The bias sets the threshold. The activation function makes the decision nonlinear. Learning adjusts $\mathbf{w}$ and $b$ to minimize errors on training data.