Neurons that aren't neurons

This chapter exists to remove a bad idea, so the removal comes first.

An artificial neuron is this: take some numbers in. Multiply each by a knob of its own – the parameters from chapter 0, now with a specific job. Add the results together. Pass the total through a simple switch that, roughly, lets strong totals through and mutes weak ones. Done. Multiply, add, switch: the entire unit, describable in a sentence, buildable from a pocket calculator’s parts.

The name is a historical accident. In the 1940s a neuroscientist and a logician proposed this multiply-add-switch as a deliberately crude cartoon of a brain cell – useful for asking what such cells could compute in principle. As biology, the cartoon was already known to be wrong in most respects; real neurons are electrochemical machines of staggering intricacy whose workings remain an active science. But the arithmetic was convenient, so the arithmetic stayed, and it kept the name. Every “neural network” in the modern world is a network of the cartoon, not of the cell. Keep the paper-aeroplane rule from the standfirst within reach: the resemblance is real, and it is a resemblance of silhouette, not of substance.

What can multiply-add-switch actually do? Something surprisingly easy to state. Give the unit two numbers – a point on a map – and its verdict divides the map with one straight line: yes on this side, no on that. That is the whole repertoire of a single unit. Different knob settings tilt and slide the line, and training, exactly as in chapter 2, nudges the line about until it separates the examples as well as one line can.

Sometimes one line is plenty. But the limitation is the interesting part, and the cleanest counterexample in the field is small enough to hold in your head. Put four points on the map, one at each corner of a square. Colour the top-left and bottom-right points yes; colour the other two corners no. Now try to separate yes from no with one straight line. Any line you draw strands a point on the wrong side – try it on paper; the failure is genuinely instructive. This little arrangement is called XOR, and a single unit cannot learn it. Not slowly, not with more examples, not with cleverer training: cannot, the way no amount of practice folds a flat sheet into a sphere.

Here is what dissolves the limitation, and it is the only genuinely clever idea in this chapter. Put several units side by side, all reading the same inputs: now you have several lines on the map. Then – the trick – feed those units’ outputs into a further unit. The second unit never sees the map. It sees only the verdicts of the first rank of lines, and it weighs verdicts exactly the way the first rank weighed coordinates. It can learn things like “above line A but below line B”: not a line any more, but a corner. More units, more layers, and corners combine into curves, curves into islands and spirals and whatever the examples demand.

Notice that nothing new was invented on the way. Multiply, add, switch, repeated and arranged. The power of a deep network comes from composition – simple decisions about simple decisions – not from any unit being clever. This is also, honestly told, where the design does echo one idea from brains, which compose simple cells into elaborate wholes. The echo is architectural, and it ends there.

The payoff is best seen rather than described. The spiral – two interleaved arms of points, a shape no straight line comes near – is the classic demonstration: a small stack of units, trained by nothing but the guess-check-adjust loop you already know, finds a boundary that curls between the arms in about fifteen seconds of watching. The scenario below holds exactly that spectacle.

Before moving on, tally what this chapter did and did not say, because the gap between the two is where the popular picture goes wrong.

It did say: units that draw lines, stacked into arrangements that draw shapes, tuned by walking a loss downhill. It did not say: understands, recognises, notices, wants. No unit in the spiral network knows there is a spiral. The network as a whole does not know it either – there is no “as a whole” to do the knowing, only more multiply-add-switch. What happened is that a boundary-shaped-like-the-data made the loss small, and the walk found it.

Holding that line takes no discipline when the subject is dots on a map. It takes real discipline two chapters from now, when the same arrangement – scaled up and pointed at language – starts producing paragraphs, and every instinct you have will reach for the mind-words. The reaching is exactly why this chapter exists. Next, though, a practical matter: the machine eats numbers, and language is not numbers. Something has to bridge that.