Slopes and the chain rule

Every mechanism so far ends at the same cliff. Training means nudging knobs to shrink the wrongness number – but which way, and by how much? The primer waved at this with fog and footsteps. This chapter builds the actual instrument, and it starts smaller than you might expect: with one knob, one output, and the humblest question in mathematics.

Turn a knob a hair. Watch the output. Did it rise or fall, and how hard? That ratio – output twitch per knob twitch – is the whole idea. A volume dial has a big ratio: small turn, loud change. A car’s thermostat has a tiny one. The ratio is a sensitivity, and knowing it answers the training question for one knob instantly: if nudging up raises the wrongness, nudge down; if the ratio is large, tread gently.

Two honest observations before the main event.

First: the sensitivity is local. The volume dial’s behaviour at a whisper says nothing about its behaviour at full blast. Wherever you stand, the ratio must be re-felt there – which is why the primer’s walker stepped and re-felt, stepped and re-felt, rather than measuring once and marching.

Second: the sensitivity is cheap to reason about for simple machines. A machine that doubles its input has ratio 2 everywhere. A machine that adds 7 has ratio 1 – shifting does not amplify. The interesting machines are built by wiring simple ones together, which raises the question the chain rule answers: if I know the ratio of every part, do I know the ratio of the whole?

Wire two machines in a chain: your knob feeds machine A, whose output feeds machine B. Nudge the knob. A’s output twitches by A’s ratio. That twitch is B’s input twitch, so B’s output twitches by its ratio applied to what arrived. Two amplifications in series – and amplifications in series multiply. Knob-to-end sensitivity: A’s ratio times B’s ratio. Gears on a bicycle, volume knobs feeding amplifiers, exchange rates through two currencies: the same multiplication, every time.

That is the chain rule, and it is the entire theorem. A chain of ten machines? Multiply ten ratios. The overall sensitivity of a long pipeline is the product of the local sensitivities along the path. Simple to the point of feeling like an accounting identity – and it is the fact the whole modern field stands on.

Now stand back and look at what you are holding.

A network’s wrongness sits at the end of a long chain: knob → layer → layer → prediction → wrongness. You want the sensitivity of the end to every knob – millions of them. The chain rule says each one is a product of local ratios along its path. And here is the industrial insight: those paths overlap enormously. The ratios near the end of the chain appear in every knob’s product, so compute them once, sweep backwards through the machine, and reuse them for every knob along the way. One cheap backward sweep, and the machine has felt every knob at once.

That sweep has a name you have heard: backpropagation. It is not a learning algorithm, not a mystery, and not intelligence. It is the chain rule, organised so nothing is computed twice. The Neural Networks course walks the bookkeeping in full; nothing in it will be new in kind, only in arrangement.