
Silicon, electricity, linear algebra: that is what a single neuron costs. The question is what you do with several, because a neuron on its own decides very little: its power comes from stacking. Neurons are arranged in layers, parallel rows whose neurons do not talk to one another, but each of which receives what the previous one has produced. The first layer receives the raw data, the points of an image (its pixels) or the words of a sentence, provided they have first been converted into numbers. The last layer gives the answer.
In between sit the hidden layers, so called because nothing they compute can be seen directly from outside. Remember that word, “hidden”. Here it is a mere convenience of vocabulary, but for a long time it named a wall: for nearly twenty years, nobody knew how to train these middle layers.

What flows: numbers, and nothing else
For a network sees nothing, in the literal sense. It can only do what you watched the neuron do in the previous episode: multiply, add, compare. So no image ever goes into a network; only numbers go into neurons, one number per input neuron. For a greyscale image the conversion needs no trick at all, because the image is already numbers: once decompressed, an image file is nothing but a grid of points, each carrying a brightness value, from pure black to pure white. The grid is unrolled into one long queue, the way you would read a chessboard square by square, and each value is plugged into a neuron. A colour image gives three values per point, red, green and blue, so three times as many numbers to plug in. No magic, just copying.

Must every image then be the same size? For the network described here, yes, with no way out. The number of input neurons is fixed once and for all, when the architecture is drawn. An input layer of seven hundred and eighty-four neurons expects seven hundred and eighty-four numbers, no more and no fewer: an image thirty points wide brings nine hundred, and a hundred and sixteen of them have nowhere to go. The same goes for colour: a network built for three values per point will not know what to do with a greyscale image, and vice versa.

The workaround is as crude as it sounds: everything is brought to the same format before it goes in. You resize, you crop, you pad the edges. And you pay the corresponding price, which is not trivial: a panoramic photograph forced into a square loses its sides, an off-centre subject leaves the frame, proportions get distorted. This constraint will be lifted in stages: in 2024, Sora, OpenAI’s video generator, will be trained on images of every size and every proportion.

Count the inputs of a real network. Take the textbook case of the field, the MNIST database of handwritten digits, released in 1998 by Yann LeCun, Corinna Cortes and Christopher Burges and since then the benchmark of every textbook: small images twenty-eight points wide, in greyscale, where each point runs from 0 for black to 255 for white. Twenty-eight times twenty-eight is seven hundred and eighty-four. The first layer of these networks therefore has exactly seven hundred and eighty-four neurons, one per point of the image, and it is customary to bring each value between 0 and 1 on the way, by dividing it by 255, so that all the incoming numbers are of the same order of magnitude. At the other end, ten output neurons, one per possible digit.
The 1989 network we will meet again soon worked on even smaller images, sixteen points wide: the principle is identical, only the count changes. In between, the hidden layers. That is a complete network, described from end to end.

Words are another matter entirely. Where the image was content with copying, language held out for decades, for one reason: a word is not a number, and resembles no number. Giving it one at random, the way you would number files, leads nowhere at all: it took until the 2010s to find the right way to do it, and we will come to that in due course. Remember the general rule: whatever you want to feed a network, image, sound, sentence or sensor reading, you must first translate it into a list of numbers. That translation is not a technical detail before the story begins; it is one of its main characters.

One line, one weight: counting a real network
Count with the diagram. Each line connects a neuron to a neuron in the next layer, and each carries a number: its weight. It decides how much the signal travelling along that line counts in the sum of the neuron it reaches. Let us pick one of these lines and give it the weight 0.80. We will meet it again further on: it is on this weight that we will watch training correct a network.

Counting the weights therefore comes down to counting the lines, and a multiplication is enough. Every neuron in a layer is connected to every neuron in the next: a neuron placed in front of a layer of 6 sends out 6 lines, and 4 neurons send out 4 × 6 = 24.

The network in the diagram lines up four layers of 4, 6, 6 and 2 neurons. The same multiplication applies between each pair of neighbouring layers: 24 lines between the first two, 6 × 6 = 36 between the next two, 6 × 2 = 12 between the last two, 72 weights in all. The biases, introduced a little further on, will add a few more without changing the order of magnitude.

Measure the gap with what is built today. In 2020, GPT-3, the OpenAI network trained to write text, has a hundred and seventy-five billion adjustable numbers, almost all of them weights: nearly two and a half billion times the seventy-two in our drawing. Yet they are counted the same way, layer after layer.
This count is no classroom exercise. In 1988, at Carnegie Mellon University, Dean Pomerleau plugs a camera into a network of this shape and lets it drive. His system, ALVINN, looks at the road through a retina of thirty by thirty-two units, each receiving the intensity of the blue in the matching patch of the image, because blue is where the road stands out best from its surroundings. A second retina, of eight by thirty-two, is fed by a laser rangefinder. In all, the network receives one thousand two hundred and seventeen numbers.
Do the sum, and it does not come out round. Nine hundred and sixty units for the video, two hundred and fifty-six for the laser: one thousand two hundred and sixteen. One is missing, and that lone unit deserves a closer look. It carries a single piece of information, the most basic there is: in the previous image, was the road lighter or darker than its edges? And it is the network itself that produces this answer, at its output; it is fed back to the input at the next step. A one-number memory, looped from output back to input.
The count now holds, and you can redo it. One thousand two hundred and seventeen inputs, twenty-nine hidden neurons, forty-six outputs: forty-five encode the curve of the bend ahead, the last is that intensity the loop will carry back to the input. One thousand two hundred and seventeen times twenty-nine, plus twenty-nine times forty-six: thirty-six thousand six hundred and twenty-seven weights.

Then came the test of a real road. The network had never seen a photograph: it had learned from twelve hundred road images generated by computer. It was nonetheless installed aboard the NAVLAB, Carnegie Mellon’s test vehicle, a Chevrolet van loaded with computers, a camera and a laser rangefinder. On a sunny autumn day, ALVINN drove it four hundred metres along a wooded path on campus, at half a metre per second, the pace of a slow walker. Another Carnegie Mellon team achieved the same accuracy twice as fast, but with an image-processing program developed over months; once the synthetic images were ready, the network, for its part, trained in about half an hour. Pomerleau, in his 1988 paper, draws the lesson: it was the data, not the programmer, that decided what to look at in the image. Twenty-four years before the 2012 car, someone had already taken the other path.

What is learned: the bias, then the parameters
One debt remains, run up in the previous episode. When you ran a neuron by hand, the instruction said: the threshold is set at 1. Set by whom, and why 1? The neuron’s character depends on that number. A high threshold makes a demanding judge, who requires overwhelming evidence before saying yes; a low threshold makes an indulgent judge, who gets carried away at the first clue. Nothing obliges every neuron in a network to share the same temperament, and above all, it is not for an engineer to decide it on their behalf: that would be going back to the 2012 car.
Check that a threshold changes what the neuron does. Take a neuron with two inputs, both weights equal to 1, and inputs worth 0 or 1. Set the threshold at 1.5: it says yes only if the sum reaches 1.5, so only when both inputs are 1. This neuron says “and”. Now, without touching a single weight, lower the threshold to 0.5: a single input at 1 is now enough. The same neuron, with the same weights, now says “or”. The threshold does not set the strength of the answer; it decides what the neuron computes.

For learning to set this temperament, it has to be put within its reach. Yet the learning rule, which we will see at work, corrects the numbers that count in the sum: the weights. The threshold is not in the sum; it waits beside it, in the comparison. Hence a trick, one of the most economical in the history of the field: the threshold is fixed once and for all, and a number specific to each neuron is added to the sum just before comparing. That number is called the bias.
Check that the bias does the threshold’s job. Go back to the “and” neuron: weights 1 and 1, threshold 1.5. With a bias of 0, nothing changes: only the sum 1 + 1 reaches 1.5. Now give it a bias of 1, without touching the threshold. The four sums become 1, 2, 2 and 3, and three of them clear 1.5: the neuron says “or”, exactly as when the threshold was lowered to 0.5. Adding 1 to the sum amounts to lowering the threshold by 1. The neuron’s temperament has become a number like the others, which training will adjust by the same mechanism as the weights, without any human having to decide it.

McCulloch and Pitts’s neuron, in 1943, already compared its sum with a threshold, but that threshold was set by hand, case by case. Rosenblatt’s perceptron turns it into an ordinary weight, by attaching it to a dummy input that is always worth 1: since this input never varies, the weight attached to it does nothing but shift the sum, and the learning rule can correct it like any other. The ADALINE of Bernard Widrow and Marcian Hoff, at Stanford in 1960, describes this setting in the same terms: an input permanently connected to a source worth 1, whose gain varies a constant added to the sum. The mechanism is there, then, and dated. The word “bias” is not yet: the 1960 paper speaks of gains, and calls this one “the level”.

That machine could be looked at. The ADALINE fitted, its authors write, into the volume of a lunch pail. A pattern was presented to it by flipping sixteen switches arranged in a square, a seventeenth switch gave the expected answer, and the error was read on a dial. The operator then turned the settings by hand, all by the same amount, each in the direction that reduced the error. Count with them: sixteen switches, and seventeen settings. The seventeenth has no switch because it expects no data: it is the bias.
One clarification, to avoid the worst misunderstanding in this series: this bias has nothing to do with prejudice. It is an offset, like setting the zero on a kitchen scale: turning the knob shifts every weighing by the same amount, without changing anything about what is being weighed.
We now need a word for everything the network learns, because it is not only the weights. Weights and biases together are the network’s parameters: the collective name for all the values that training adjusts, one by one, so that the machine answers correctly. The weight is therefore one kind of parameter, the most numerous, but not the only one. Take care not to confuse it with another setting: the number of layers is not a learned parameter but an architectural choice fixed in advance, what is called a hyperparameter. The architect decides how many storeys there are and how wide they are; training decides only what happens inside. When a model’s size is announced, it is the learned parameters that are counted, never the number of layers.

And that is exactly what a model is: on one side a gigantic file of numbers, the learned parameters; on the other a small configuration file, the wiring plan, which says how many layers, how wide, which operations and in what order to chain them. The first weighs tens or hundreds of gigabytes, the second fits in a handful of lines. Together, they are enough to run the model. The small network in this episode would thus fit in eighty-six numbers, its seventy-two weights and its fourteen biases, and one line that says 4, 6, 6, 2.

Nobody wrote these numbers by hand. The question is what does write them: when a network with several layers makes a mistake, nothing says which of its weights is at fault, and that is the question that will have to be solved for stacking to be of any use.
Sources
What flows: numbers, and nothing else
- Gradient-based learning applied to document recognition · Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner, Proceedings of the IEEE, vol. 86, no. 11, 1998, pp. 2278 to 2324, doi 10.1109/5.726791: the paper that presents the MNIST database and its images of twenty-eight points by twenty-eight; peer-reviewed
- Backpropagation Applied to Handwritten Zip Code Recognition · Yann LeCun et al., Neural Computation, vol. 1, no. 4, 1989: the 1989 network and its images of sixteen points by sixteen; peer-reviewed
- Video generation models as world simulators · OpenAI, 15 February 2024: Sora’s training on videos of varying durations, resolutions and proportions; company technical report, not peer-reviewed
One line, one weight: counting a real network
- Language Models are Few-Shot Learners · Tom B. Brown et al., OpenAI, arXiv preprint, May 2020: GPT-3 and its one hundred and seventy-five billion parameters
- ALVINN: An Autonomous Land Vehicle in a Neural Network · Dean Pomerleau, proceedings of the NIPS 1988 conference: the network’s architecture, its unit counts, training on synthetic images and the test on the NAVLAB; peer-reviewed
What is learned: the bias, then the parameters
- Adaptive Switching Circuits · Bernard Widrow and Marcian E. Hoff, IRE WESCON Convention Record, vol. 4, August 1960, pp. 96 to 104: page 96, the input permanently connected to a source worth 1; page 97, the machine the size of a lunch pail, its sixteen switches and its seventeen gains













