Chapter poster: on the left, a Nevada road at sunset; on the right, a Google self-driving car in a street; bottom left, the chapter title and a line from the text.
Left, a Nevada road at sunset: Route 50, photographed in 2020. Right, a Google self-driving car, a Lexus fitted with its sensors, photographed in 2014, two years after the February 2012 course the text starts from. Amine Abassir, Nevada Highway 50 Sunset, 15 May 2020. CC BY-SA 4.0. Source: Wikimedia Commons. Anthony Quintano, Google Self Driving Car (52639793867), 2 April 2014. CC BY 2.0. Source: Wikimedia Commons. Composition, crop and grading by Shy Robotics; the whole is released under the same licence, CC BY-SA 4.0.

On 21 February 2012, I shared on Shy Robotics a course by Sebastian Thrun (in French), the Stanford professor whose robot Stanley had crossed the Nevada desert, before he went on to launch Google’s self-driving car. On the syllabus: particle filters, Kalman filters, the A* algorithm, the PID controller. The textbook of an era, one in which building an intelligent machine meant writing yourself, equation after equation, every rule it would follow.

I practised that textbook. Two years earlier, in 2010, I was leading Aquatis, an autonomous underwater robot our team had entered in the European SAUC-E competition, held at a NATO base. Kalman, PID, mapping, computer vision: we were living exactly what Thrun taught. Neural networks, programs that learn from examples, we knew about and played with: I was training a small C library called FANN to recognise handwritten digits, for instance. But they were confined to a few very particular uses, slow and hungry for computing power. Putting them on board a robot under the tyranny of real time seemed unreasonable: we already had to split the smallest calculation between a slow computer, for thinking, and fast microcontrollers, for reflexes, or the machine decided too late. The idea that a single big box of neurons might one day do everything, and do it better, occurred to nobody around me.

That was the state of the art, and it was brilliant. All the ingenuity of the profession went into polishing these hand-written methods, into squeezing out a few points of accuracy from one year to the next. And while we were perfecting our filters, a tipping point was building that almost nobody saw coming.

It came in the autumn of 2012, at the ImageNet image recognition competition, where the world’s best teams had been fighting for years to gain half a point.

The 2012 competition had three tasks. The main one, classification, covered a thousand categories of objects: about 1.2 million images whose category was given, for learning, 50,000 for tuning, and 100,000 test images whose answers were kept secret. For each test image, a program returned at most five guesses, from most to least likely, and the image counted as a success if the right category was among them. The error rate is the share of images missed.

The second task, localisation, also required drawing a box around the object in the image: the proposed box had to overlap the one drawn by the annotators by more than half. The third, more fine-grained, covered about a hundred dog breeds. Teams had until 30 September to submit their answers; the results were unveiled on 12 October, in Florence.

Three panels drawn in pencil. First panel, a leopard: the photo yields five guesses, jaguar, cheetah, leopard, lynx, wildcat, and leopard, highlighted in green, is among them: correct. Second panel, a car grille: the five guesses, convertible, limousine, bumper, cab, wheel rim, do not include it: missed. Third panel, localise: on a photo, a green box overlaps a dotted blue reference box by more than half.
The first two tasks of ImageNet 2012, with illustrative examples. Classify: five guesses per image, and the image counts as a success if the right answer, highlighted in green, is among them. Localise: the proposed box, in a solid line, must overlap the reference box, dotted, by more than half.

That year, a program from Toronto called AlexNet blew the first task’s rankings apart. Where the runner-up topped out at a 26.2% error rate, it fell to 15.3%. A gap of nearly eleven points, unheard of in the history of the competition. And AlexNet was not one more hand-tuned method: it was a neural network, that old idea the field barely looked at. In less than two years, Google, Facebook, Microsoft and Baidu all rushed into deep learning, the name given to large neural networks. The textbook I had just shared was not wrong; in a single evening, it had become a historical document.

That same year, Geoffrey Hinton and two of his students, Alex Krizhevsky and Ilya Sutskever, founded a neural network company, DNNresearch; Google bought it in March 2013. At the end of that same year, 2013, Google announced the purchase of eight robotics companies, among them Boston Dynamics and its quadrupeds, Schaft and its humanoid, and Industrial Perception and its vision for handling arms, and put the whole lot under Andy Rubin, the father of Android. Those machines walked, jumped and grasped; their skill was programmed, not learned, exactly like that of the February 2012 car. Nine months apart, the same company had therefore bought software that learned to recognise what it saw, and robots that knew how to move. The two purchases fitted together exactly: the software lacked a body, and the robots lacked the ability to learn. It was no longer unreasonable to imagine what came next: a human-shaped robot that would learn by example instead of being programmed move by move.

Atlas, the humanoid robot of Boston Dynamics and DARPA, seen from the front in 2013
Atlas, the humanoid robot built by Boston Dynamics with DARPA, photographed in 2013. In the year Google bought the company, these machines already walk and grasp; nothing they do has been learned. DARPA, Atlas, front view, 2013. Public domain. Source: Wikimedia Commons

The most dizzying part lies elsewhere. The idea that had just swept everything before it was not new: it had been asleep for more than half a century, invented and then buried twice by the scientific community itself. Here is how it came back from the dead, what it swept away as it went, what still resists it, and what you need to understand about it in order to use it.

Four brilliant methods, and not one that learns

Let us go back to Sebastian Thrun’s course, because it says everything about the spirit of the time. Each of its building blocks was a small marvel of human ingenuity, and none of them learned anything. The particle filter located the car by scattering thousands of position hypotheses across the map, like a handful of rice grains thrown at random, then eliminating one by one those the sensors contradicted, until the cloud closed in around the only plausible point. The Kalman filter tracked the other vehicles by deciding, at every moment, between what the sensor reported and what the physics of motion predicted. The A* algorithm laid out the best route, and the PID controller held the steering wheel, constantly correcting the gap from the planned path.

In each of these methods, an engineer had written the rule beforehand. How to weigh a sensor, how to estimate a trajectory, how to correct an error: everything was thought out, formulated and coded by a human. The machine only carried out, admirably, a line of reasoning that was not its own. Over the seven weeks of the course, a single session touched on machine learning, and even then only to sort sensor points. That shows how far the idea of a machine forging its own rules remained, for us, a laboratory curiosity rather than a field tool.

That laboratory curiosity, however, was about to sweep everything away. There was another path, the exact opposite of that car: a machine that did not wait for an engineer to dictate its rules, but learned them on its own, from examples. AlexNet was a neural network, and its victory in the autumn of 2012 swung a whole branch of research towards the approach Thrun’s car still ignored.

But the idea was not new. It had already been invented, buried, revived, then buried a second time, over nearly seventy years. Before telling that eventful story, we need to understand what a neural network actually is, and why the name is misleading.

Two pencil-drawn timelines. At the top, from 1943 to 2012: the neuron model, the perceptron in 1958, the first burial in 1969, backpropagation in 1986, convolution in 1989 and AlexNet in 2012, with two hatched bands for the winters from 1969 to 1986 and from 1990 to 2012. At the bottom, from 2012 to 2026: AlexNet, embeddings in 2013, the Transformer in 2017, GPT-3 in 2020, ChatGPT in 2022, reasoning models in 2024 and world models in 2026.
The two rows are not on the same scale, deliberately: sixty-nine years take up the same width here as fourteen. Comparing distances from one row to the other would mislead; comparing the number of milestones would not.

Why these neural networks have nothing biological about them

Before going further, we need to clear up the most stubborn misunderstanding in the field. The artificial neuron resembles a human neuron in name only, a name inherited from a metaphor of 1943. That year, the neurophysiologist Warren McCulloch and the logician Walter Pitts published a deliberately crude mathematical model of the nerve cell. It comes down to three operations, and they are still those of today’s models: receive several numbers, multiply each by a coefficient (set identically for all excitatory inputs), add everything up, and apply a function that decides the output.

The meeting of the two men who laid that first stone reads like a novel. McCulloch is forty-two, he has studied logic, and for years he has been trying to describe thought as a calculation without knowing how to handle the mathematics. Walter Pitts has no degree and no roof over his head. A medical student, Jerome Lettvin, introduces them; McCulloch and his wife take the young man into their home. The partnership works so well that in 1943 they publish “A Logical Calculus of the Ideas Immanent in Nervous Activity”. Their aim was not to build a learning machine. They wanted to show that reasoning itself could be written in logic, and the neuron was only the vehicle for that demonstration.

It shows in the model, down to one detail that is the whole subject of this series: in McCulloch and Pitts’s neuron, the coefficients cannot be adjusted. All the connections that excite the neuron have exactly the same strength. A single connection that inhibits it is enough to block it, whatever the others say. The threshold at which it responds is set by hand by whoever draws the circuit. And none of it ever changes. The 1943 neuron is a piece of logic, a cleverly wired switch that computes “and”, “or” and “not”, and will compute them the same way until the end of time. It learns nothing.

So a piece is missing, and it is the piece that will drive the rest of this story: a number attached to each input, one you can turn like a dial. That number has a name, the weight, and it is what separates a circuit from a learner. On a jury where each juror gives a score, the weight is how much credit each one gets in the average: raising a juror’s credit changes the verdict without changing a single score. The activation function plays the role of the threshold: below it nothing passes, above it the signal carries on. An artificial neuron, in the form that will serve all the way to today’s models, is therefore a multiplication, an addition and a comparison. No chemistry, no action potential, no neurotransmitter.

Two twin panels drawn in pencil. On the left, the 1943 neuron of McCulloch and Pitts: three inputs each multiplied by 1, and nothing can be adjusted. On the right, the same neuron with one weight per input, 2.1, minus 0.7 and 0.4, framed in red. The threshold is the same on both sides.
McCulloch and Pitts were not trying to fill this gap. Their subject was a philosophical question, not a learning machine: what is missing on the left was not missing for them.

Which leaves the question that drives the rest of the series: who turns these dials? In 1943, nobody but the engineer’s hand. It would take six years for a psychologist to venture the hypothesis that they might adjust themselves, and fifteen for a machine to actually do it. Let us first look at the object itself, once its weights are set, with an example.

A pencil-drawn artificial neuron: three pixels as inputs, 0.9 times 2.1, 0.1 times minus 0.7 and 0.5 times 0.4; the sum is 2.02, it is above the threshold of 1, and the neuron answers yes.
The artificial neuron in the form that will serve all the way to today’s models: one weight per input, an addition, a threshold. The figures are illustrative. What is accurate: McCulloch and Pitts’s 1943 model did have this framework, but its weights could not be adjusted.

Take the three numbers in the diagram. Multiply each input by its weight: 0.9 × 2.1 gives 1.89; 0.1 × (minus 0.7) gives minus 0.07; 0.5 × 0.4 gives 0.20. Add them up: 1.89 minus 0.07 plus 0.20 equals 2.02. The threshold was set at 1. Since 2.02 is above 1, the neuron says yes. You have just run, by hand, the basic unit of all modern artificial intelligence. A giant model does nothing else but this operation, thousands of billions of times a second. To measure the gap: in 1969, the fastest machine in the world, the CDC 7600, which would remain so until 1975, carried out around ten million operations a second.

Neural computation does not fit on an ordinary processor

What the neuron asks of a machine comes down to three elementary moves, always the same. In a neural network, thousands of neurons repeat that same small calculation, each on its own numbers, with no need for the neighbour’s result. These operations are independent, so they can all be done at the same time.

That rules out the ordinary processor, the CPU (central processing unit). A CPU has a few cores, the units that carry out the calculations, very fast and very clever, designed to chain varied tasks one after another: an excellent conductor, but one who plays the instruments one at a time. Neural computation does not call for sequential cleverness; it calls for doing the same trivial operation millions of times, all together. That is exactly what the graphics card and its processor, the GPU (graphics processing unit), had been built for: displaying a video game means computing the colour of millions of pixels simultaneously, the same move repeated everywhere. A GPU therefore trades the CPU’s few gifted cores for thousands of modest cores all working side by side. The coincidence is almost comic: the dream tool of artificial intelligence had been perfected, for years, to run shooting games.

Two components seen from above, drawn in pencil. On the left, a processor and its four large cores. On the right, a graphics card and 128 small cores. A highlighted note recalls that, in reality, a few cores face thousands.
The drawing cannot show what matters most: a processor core is far more complex than a graphics card core, and that does not show in the size of a square. The card only wins on calculations that do not need one another.

Let us measure the reversal, because it is one of the quietest revolutions in the recent history of computing. The algorithms of the 2012 car, Kalman, A*, PID, were perfectly at home on an ordinary processor: a series of logical steps, each resting on the one before, exactly the sequential regime the CPU had been designed for since the 1970s. For half a century, the whole software industry was built on that assumption: programming means writing a recipe that the machine follows step by step. The neural network breaks that assumption. It does not follow a recipe; it crushes mountains of multiplications in parallel, and so it calls for hardware of an entirely different kind.

The mechanism is easier to see in reverse, on a calculation that does not lend itself to it. Give a Kalman filter a thousand cores: it will go no faster. It can only estimate the vehicle’s position at a given moment once it has settled the position at the moment before, so the steps queue up one behind another. The calculation for a single step fits in tables of a few numbers: enough to keep one core busy, not a thousand. This failure teaches the rule better than success would: what decides is not the amount of work, but its shape. Tasks that ignore each other can be shared out; a task made of steps that hold each other by the hand cannot, whatever the machine.

One caution: later in this series, we will come to the layers of a neural network. Independence holds within one of those layers, not from one layer to the next, since each waits for the result of the one before. A single layer already provides enough to keep thousands of cores busy.

Overnight, or nearly, the component that mattered was no longer the processor but the graphics card. A part associated with teenagers and shooting games was about to become the most fought-over strategic resource on the planet. Changing the algorithm did not only change the way programs were written; it redistributed value across the whole computing industry, from software to hardware.

Since then, the race for silicon has never stopped. Today we know how to make chips even more specialised than the GPU, cut out for these calculations alone: Google’s TPUs (tensor processing units), the NPUs (neural processing units) in our phones. But the GPU keeps a lead that has nothing to do with raw performance: it was already there, mass-produced and easy to program, when the others still had to be invented. All this extravagance of engineering for a reason that remains, at bottom, disarmingly simple: multiply numbers, add them up, start again billions of times. Silicon, electricity, linear algebra.

Sources

February 2012: a car that learned nothing

Why these neural networks have nothing biological about them

Neural computation does not fit on an ordinary processor

  • CDC 6600’s Five Year Reign · Computer History Museum: Control Data’s dominance and the handover to the CDC 7600 in 1969; museum notice

LAISSER UN COMMENTAIRE