Gradients and descent

Chapter 3 left the machine able to feel, for every knob, whether nudging it up makes the wrongness rise or fall, and how hard. For a million-knob machine that is a million little verdicts. This chapter is about reading them all at once – and discovering they were never a million verdicts at all, but one.

Collect the sensitivities into a single list, one entry per knob. By chapter 1’s habit, a list of numbers is an arrow in knob-space. So the million verdicts assemble into one arrow, and the question “which way should I nudge everything?” becomes “which way does this arrow point?”

The answer is the cleanest fact in the course: it points exactly uphill – the direction in knob-space that increases the wrongness fastest. And the training rule falls out before the section break: walk the other way.

The procedure, in full: feel the arrow, take one small step directly against it, and – because chapter 3 warned that sensitivities are local – stop and feel again. Step, feel, step, feel. This is the primer’s foggy walk with the fog explained: the machine never sees the landscape because seeing it whole would mean trying every knob setting; it only ever holds one arrow, valid near where it stands.

The method’s proper name is steepest descent – each step takes the locally steepest downhill direction – and its everyday name is gradient descent. It is the procedure this site is named after, and the naming is meant literally: every course that follows, whatever it teaches, is teaching refinements, consequences or applications of this walk. When the primer trained a line to separate two clouds of dots, this walk chose every nudge. It is doing the same, at scale, inside every model you have ever heard of.

How big should the step be? The arrow is honest only near where you stand – walk too far along it and you are navigating by a slope that no longer describes the ground. Too timid, and training crawls: real budgets, real electricity, real weeks. Too bold, and each stride overshoots the valley and lands higher than it took off; the wrongness climbs while the machine walks downhill with total conviction, until the run is dead.

That failure is worth watching once with your own eyes – it is the most legible catastrophe in the field, and the first scenario below runs it live. The second stages a fairer contest: three walkers on the same landscape, identical in everything but stepping temperament. The differences you will see are the entire subject of “optimisers”, refined – but not replaced – in every modern training run.

Two clauses, stated plainly because this course exists to state them.

The walk finds a valley, not the valley. It follows local slope, so it settles wherever downhill runs out – and a deeper valley may sit unvisited beyond the next ridge. Nobody has repealed this. The working truth of the field is startling but honest: for the enormous, high-dimensional landscapes of real models, the valleys this walk finds are almost always good enough, and nobody can fully explain why. “Steepest” is a local word, and everything downstream inherits that.

And the walk needs the ground to be feelable – smooth enough that nudge-and-measure means something. Machine learning’s insistence that everything be built from smooth pieces (remember the primer’s gentle switches) is not aesthetic; it is this walk demanding terrain it can sense.