From Perceptron to Network
A perceptron is the simplest neuron: it multiplies each input by a weight, sums them, adds a bias, and applies an activation:
z = w₁·x₁ + w₂·x₂ + … + b
a = activation(z)One neuron can only draw a straight decision boundary. The leap of deep learning: stack many neurons into layers, so the network learns increasingly abstract features — edges, then shapes, then objects.
The Forward Pass
Information flows left to right. Each layer computes a weighted sum of the previous layer’s outputs, then applies a non-linearity. For a 3→4→1 network:
- The input layer forwards the raw features.
- The hidden layer computes 4 weighted sums and activates them.
- The output layer produces a single prediction.
The neural-network visualizer on this page steps through this exact sequence with real numbers.
Activation Functions
The non-linearity is what makes stacks of layers more powerful than one big linear model.
| Function | Output range | Used for |
|---|---|---|
| Sigmoid | (0, 1) | Final probability |
| ReLU | [0, ∞) | Hidden layers (default) |
| Tanh | (−1, 1) | Hidden layers |
Without a non-linearity, two stacked linear layers collapse into one — no added capacity.
Loss: How Wrong Are We?
The loss compares the prediction ŷ to the label y. Cross-entropy punishes confident mistakes heavily:
L = −[ y·log(ŷ) + (1−y)·log(1−ŷ) ]A single loss number summarizes error for one example (or the average over a batch). The goal of training is to make this number as small as possible.
Backpropagation and Gradient Descent
Backpropagation computes how much each weight contributed to the loss, using the chain rule to push the error backwards through the network — from output to hidden layer to input.
Gradient descent then nudges every weight downhill on the loss surface:
w ← w − η · ∂L/∂w| Learning rate η | Effect |
|---|---|
| Too large | Overshoots, loss explodes, training diverges |
| Too small | Converges, but painfully slowly |
| Just right | Fast, stable convergence |
Training Is Iteration
One epoch = one full pass over the training set. Training loops forward pass → loss → backprop → update across thousands of batches. You monitor loss on the validation set, not the training set, to catch overfitting, and you stop when validation performance stops improving.
When It’s the Right Tool
Choose a neural network when the signal is complex, high-dimensional, or perceptual — images, audio, natural language. For small, tabular datasets with clean features, a gradient-boosted tree is often simpler, faster to train, and just as accurate. Reach for deep learning because the problem needs it, not because it’s fashionable.
Practice Trajectory
- Compute the forward pass for the visualizer’s 3→4→1 network by hand.
- Explain why ReLU is preferred over sigmoid in hidden layers.
- Trace the chain rule path that lets one weight in the input layer receive a gradient.
- Set η too large in a mental experiment and predict what the loss curve looks like.
- Compare when you would choose a tree model instead of a neural network.