
On 21 February 2012, I shared on Shy Robotics a course by Sebastian Thrun (in French), the Stanford professor whose robot Stanley had crossed the Nevada desert, before he went on to launch Google’s self-driving car. On the syllabus: particle filters, Kalman filters, the A* algorithm, the PID controller. The textbook of an era, one in which building an intelligent machine meant writing yourself, equation after equation, every rule it would follow.
I practised that textbook. Two years earlier, in 2010, I was leading Aquatis, an autonomous underwater robot our team had entered in the European SAUC-E competition, held at a NATO base. Kalman, PID, mapping, computer vision: we were living exactly what Thrun taught. Neural networks, programs that learn from examples, we knew about and played with: I was training a small C library called FANN to recognise handwritten digits, for instance. But they were confined to a few very particular uses, slow and hungry for computing power. Putting them on board a robot under the tyranny of real time seemed unreasonable: we already had to split the smallest calculation between a slow computer, for thinking, and fast microcontrollers, for reflexes, or the machine decided too late. The idea that a single big box of neurons might one day do everything, and do it better, occurred to nobody around me.
That was the state of the art, and it was brilliant. All the ingenuity of the profession went into polishing these hand-written methods, into squeezing out a few points of accuracy from one year to the next. And while we were perfecting our filters, a tipping point was building that almost nobody saw coming.
It came in the autumn of 2012, at the ImageNet image recognition competition, where the world’s best teams had been fighting for years to gain half a point.
The 2012 competition had three tasks. The main one, classification, covered a thousand categories of objects: about 1.2 million images whose category was given, for learning, 50,000 for tuning, and 100,000 test images whose answers were kept secret. For each test image, a program returned at most five guesses, from most to least likely, and the image counted as a success if the right category was among them. The error rate is the share of images missed.
The second task, localisation, also required drawing a box around the object in the image: the proposed box had to overlap the one drawn by the annotators by more than half. The third, more fine-grained, covered about a hundred dog breeds. Teams had until 30 September to submit their answers; the results were unveiled on 12 October, in Florence.
That year, a program from Toronto called AlexNet blew the first task’s rankings apart. Where the runner-up topped out at a 26.2% error rate, it fell to 15.3%. A gap of nearly eleven points, unheard of in the history of the competition. And AlexNet was not one more hand-tuned method: it was a neural network, that old idea the field barely looked at. In less than two years, Google, Facebook, Microsoft and Baidu all rushed into deep learning, the name given to large neural networks. The textbook I had just shared was not wrong; in a single evening, it had become a historical document.
That same year, Geoffrey Hinton and two of his students, Alex Krizhevsky and Ilya Sutskever, founded a neural network company, DNNresearch; Google bought it in March 2013. At the end of that same year, 2013, Google announced the purchase of eight robotics companies, among them Boston Dynamics and its quadrupeds, Schaft and its humanoid, and Industrial Perception and its vision for handling arms, and put the whole lot under Andy Rubin, the father of Android. Those machines walked, jumped and grasped; their skill was programmed, not learned, exactly like that of the February 2012 car. Nine months apart, the same company had therefore bought software that learned to recognise what it saw, and robots that knew how to move. The two purchases fitted together exactly: the software lacked a body, and the robots lacked the ability to learn. It was no longer unreasonable to imagine what came next: a human-shaped robot that would learn by example instead of being programmed move by move.
The most dizzying part lies elsewhere. The idea that had just swept everything before it was not new: it had been asleep for more than half a century, invented and then buried twice by the scientific community itself. Here is how it came back from the dead, what it swept away as it went, what still resists it, and what you need to understand about it in order to use it.
Four brilliant methods, and not one that learns
Let us go back to Sebastian Thrun’s course, because it says everything about the spirit of the time. Each of its building blocks was a small marvel of human ingenuity, and none of them learned anything. The particle filter located the car by scattering thousands of position hypotheses across the map, like a handful of rice grains thrown at random, then eliminating one by one those the sensors contradicted, until the cloud closed in around the only plausible point. The Kalman filter tracked the other vehicles by deciding, at every moment, between what the sensor reported and what the physics of motion predicted. The A* algorithm laid out the best route, and the PID controller held the steering wheel, constantly correcting the gap from the planned path.
In each of these methods, an engineer had written the rule beforehand. How to weigh a sensor, how to estimate a trajectory, how to correct an error: everything was thought out, formulated and coded by a human. The machine only carried out, admirably, a line of reasoning that was not its own. Over the seven weeks of the course, a single session touched on machine learning, and even then only to sort sensor points. That shows how far the idea of a machine forging its own rules remained, for us, a laboratory curiosity rather than a field tool.
That laboratory curiosity, however, was about to sweep everything away. There was another path, the exact opposite of that car: a machine that did not wait for an engineer to dictate its rules, but learned them on its own, from examples. AlexNet was a neural network, and its victory in the autumn of 2012 swung a whole branch of research towards the approach Thrun’s car still ignored.
But the idea was not new. It had already been invented, buried, revived, then buried a second time, over nearly seventy years. Before telling that eventful story, we need to understand what a neural network actually is, and why the name is misleading.
Why these neural networks have nothing biological about them
Before going further, we need to clear up the most stubborn misunderstanding in the field. The artificial neuron resembles a human neuron in name only, a name inherited from a metaphor of 1943. That year, the neurophysiologist Warren McCulloch and the logician Walter Pitts published a deliberately crude mathematical model of the nerve cell. It comes down to three operations, and they are still those of today’s models: receive several numbers, multiply each by a coefficient (set identically for all excitatory inputs), add everything up, and apply a function that decides the output.
The meeting of the two men who laid that first stone reads like a novel. McCulloch is forty-two, he has studied logic, and for years he has been trying to describe thought as a calculation without knowing how to handle the mathematics. Walter Pitts has no degree and no roof over his head. A medical student, Jerome Lettvin, introduces them; McCulloch and his wife take the young man into their home. The partnership works so well that in 1943 they publish “A Logical Calculus of the Ideas Immanent in Nervous Activity”. Their aim was not to build a learning machine. They wanted to show that reasoning itself could be written in logic, and the neuron was only the vehicle for that demonstration.
It shows in the model, down to one detail that is the whole subject of this series: in McCulloch and Pitts’s neuron, the coefficients cannot be adjusted. All the connections that excite the neuron have exactly the same strength. A single connection that inhibits it is enough to block it, whatever the others say. The threshold at which it responds is set by hand by whoever draws the circuit. And none of it ever changes. The 1943 neuron is a piece of logic, a cleverly wired switch that computes “and”, “or” and “not”, and will compute them the same way until the end of time. It learns nothing.
So a piece is missing, and it is the piece that will drive the rest of this story: a number attached to each input, one you can turn like a dial. That number has a name, the weight, and it is what separates a circuit from a learner. On a jury where each juror gives a score, the weight is how much credit each one gets in the average: raising a juror’s credit changes the verdict without changing a single score. The activation function plays the role of the threshold: below it nothing passes, above it the signal carries on. An artificial neuron, in the form that will serve all the way to today’s models, is therefore a multiplication, an addition and a comparison. No chemistry, no action potential, no neurotransmitter.
Which leaves the question that drives the rest of the series: who turns these dials? In 1943, nobody but the engineer’s hand. It would take six years for a psychologist to venture the hypothesis that they might adjust themselves, and fifteen for a machine to actually do it. Let us first look at the object itself, once its weights are set, with an example.
Take the three numbers in the diagram. Multiply each input by its weight: 0.9 × 2.1 gives 1.89; 0.1 × (minus 0.7) gives minus 0.07; 0.5 × 0.4 gives 0.20. Add them up: 1.89 minus 0.07 plus 0.20 equals 2.02. The threshold was set at 1. Since 2.02 is above 1, the neuron says yes. You have just run, by hand, the basic unit of all modern artificial intelligence. A giant model does nothing else but this operation, thousands of billions of times a second. To measure the gap: in 1969, the fastest machine in the world, the CDC 7600, which would remain so until 1975, carried out around ten million operations a second.
Neural computation does not fit on an ordinary processor
What the neuron asks of a machine comes down to three elementary moves, always the same. In a neural network, thousands of neurons repeat that same small calculation, each on its own numbers, with no need for the neighbour’s result. These operations are independent, so they can all be done at the same time.
That rules out the ordinary processor, the CPU (central processing unit). A CPU has a few cores, the units that carry out the calculations, very fast and very clever, designed to chain varied tasks one after another: an excellent conductor, but one who plays the instruments one at a time. Neural computation does not call for sequential cleverness; it calls for doing the same trivial operation millions of times, all together. That is exactly what the graphics card and its processor, the GPU (graphics processing unit), had been built for: displaying a video game means computing the colour of millions of pixels simultaneously, the same move repeated everywhere. A GPU therefore trades the CPU’s few gifted cores for thousands of modest cores all working side by side. The coincidence is almost comic: the dream tool of artificial intelligence had been perfected, for years, to run shooting games.
Let us measure the reversal, because it is one of the quietest revolutions in the recent history of computing. The algorithms of the 2012 car, Kalman, A*, PID, were perfectly at home on an ordinary processor: a series of logical steps, each resting on the one before, exactly the sequential regime the CPU had been designed for since the 1970s. For half a century, the whole software industry was built on that assumption: programming means writing a recipe that the machine follows step by step. The neural network breaks that assumption. It does not follow a recipe; it crushes mountains of multiplications in parallel, and so it calls for hardware of an entirely different kind.
The mechanism is easier to see in reverse, on a calculation that does not lend itself to it. Give a Kalman filter a thousand cores: it will go no faster. It can only estimate the vehicle’s position at a given moment once it has settled the position at the moment before, so the steps queue up one behind another. The calculation for a single step fits in tables of a few numbers: enough to keep one core busy, not a thousand. This failure teaches the rule better than success would: what decides is not the amount of work, but its shape. Tasks that ignore each other can be shared out; a task made of steps that hold each other by the hand cannot, whatever the machine.
One caution: later in this series, we will come to the layers of a neural network. Independence holds within one of those layers, not from one layer to the next, since each waits for the result of the one before. A single layer already provides enough to keep thousands of cores busy.
Overnight, or nearly, the component that mattered was no longer the processor but the graphics card. A part associated with teenagers and shooting games was about to become the most fought-over strategic resource on the planet. Changing the algorithm did not only change the way programs were written; it redistributed value across the whole computing industry, from software to hardware.
Since then, the race for silicon has never stopped. Today we know how to make chips even more specialised than the GPU, cut out for these calculations alone: Google’s TPUs (tensor processing units), the NPUs (neural processing units) in our phones. But the GPU keeps a lead that has nothing to do with raw performance: it was already there, mass-produced and easy to program, when the others still had to be invented. All this extravagance of engineering for a reason that remains, at bottom, disarmingly simple: multiply numbers, add them up, start again billions of times. Silicon, electricity, linear algebra.
Sources
February 2012: a car that learned nothing
- Course 1: building a self-driving car (in French) · Shy Robotics, February 2012
- Self-supervised Monocular Road Detection in Desert Terrain · Hendrik Dahlkamp, Adrian Kaehler, David Stavens, Sebastian Thrun and Gary Bradski, Robotics: Science and Systems II, August 2006: Stanley’s team describes its own race, a hundred and thirty-two miles in seven hours, and names the Nevada desert; peer reviewed
- Novel approach to nonlinear/non-Gaussian Bayesian state estimation · N. J. Gordon, D. J. Salmond and A. F. M. Smith, IEE Proceedings F, Radar and Signal Processing, volume 140, no. 2, 1993, page 107: the paper that founded the particle filter described above, the first of the four tools on the February 2012 syllabus; peer reviewed, doi 10.1049/ip-f-2.1993.0015
- What we’re driving at · official Google blog, 9 October 2010, by Sebastian Thrun, Distinguished Software Engineer: Google’s self-driving car project announced by the very man who taught the February 2012 course
- ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) · Stanford Vision Lab: the three tasks, a thousand categories, 1.2 million training images, 50,000 for validation and 100,000 for testing, at most five guesses per image, the overlap of more than 50% required for localisation, the 30 September 2012 deadline and the 12 October workshop in Florence
- ImageNet 2012 official results (ILSVRC2012) · Stanford Vision Lab: the SuperVision team (Krizhevsky, Sutskever, Hinton, University of Toronto) at 0.15315 error, the runner-up at 0.26172
- Google acquires U of T neural networks company · Sara Franca, University of Toronto news release, 12 March 2013: Google buys the company of Geoffrey Hinton, Alex Krizhevsky and Ilya Sutskever
- Google buys University of Toronto startup · CBC, 13 March 2013: this piece gives the company’s name, DNNresearch Inc., incorporated in 2012 by the same three
- Baidu Opens Silicon Valley Lab, Appoints Andrew Ng as Head of Baidu Research · Baidu press release, 16 May 2014: fourteen months later, the other end of the period; the same release dates the Institute of Deep Learning to 2013
- Yann LeCun, Chief AI Scientist, Facebook AI Research · Meta profile, undated: it establishes LeCun’s arrival at Facebook in December 2013
- On Welsh Corgis, Computer Vision, and the Power of Deep Learning · Microsoft Research blog, 14 July 2014: Project Adam, begun eighteen months earlier, in early 2013
- Boston Dynamics Now Belongs to Google · Evan Ackerman and Erico Guizzo, IEEE Spectrum, 14 December 2013, trade press: the seven robotics start-ups bought in early December, then Boston Dynamics, all placed under Andy Rubin
- Google Acquires Seven Robot Companies, Wants Big Role in Robotics · IEEE Spectrum, 4 December 2013; this piece, not the previous one, names the seven: Schaft, “a small team of Japanese roboticists who recently left the University of Tokyo”, Industrial Perception and its vision-guided arms, Meka and Redwood Robotics in San Francisco, Bot & Dolly, Autofuss and Holomni
Why these neural networks have nothing biological about them
- A Logical Calculus of the Ideas Immanent in Nervous Activity · McCulloch and Pitts, Bulletin of Mathematical Biophysics, 1943
- The assumptions of the 1943 model · University of Texas at San Antonio notes, on inhibition as a veto
- The Man Who Tried to Redeem the World with Logic · Amanda Gefter, Nautilus no. 21, 29 January 2015: magazine journalism drawing on the McCulloch Papers and Talking Nets; McCulloch’s age when they met, the introduction by Jerome Lettvin and Pitts taken into his home
Neural computation does not fit on an ordinary processor
- CDC 6600’s Five Year Reign · Computer History Museum: Control Data’s dominance and the handover to the CDC 7600 in 1969; museum notice