Getting less wrong

Chapter 1 ended on a question: which way is less wrong? The answer begins by making wrongness measurable.

Suppose the model is asked “is this a cat?” and answers with its confidence: 0.7, on a scale where 1 means certainly yes. The true answer was yes – a 1. The miss has a size: 0.3. A different setting of the knobs might have answered 0.9 and missed by 0.1. Smaller miss, better setting. Do this over every example in the pile and average the misses, and you get a single number that says how wrong the model is overall, right now, with these knobs.

The field’s name for that number is loss, and it is one of maybe six words worth carrying away from this whole primer. High loss: the predictions are far from the answers. Low loss: close. Training, which chapter 1 described as “nudge the machine to be less wrong”, can now be said exactly: adjust the knobs to make the loss smaller. Nothing about cats, nothing about language, nothing about intelligence – every kind of model you have heard of trains by shrinking a number of this kind. Once wrongness is a number, improving is arithmetic, and arithmetic can be automated.

Now the picture that organises the whole field. Imagine a model with only two knobs. Every possible setting of the pair is a place on a map – first knob east–west, second knob north–south – and at every place, the loss has some value. Let that value be the height of the ground there. Settings that predict badly are high ground; settings that predict well are valleys. Training is now a walk: start wherever your random knobs dropped you, and get downhill.

Real models have billions of knobs, not two, so the real landscape has billions of directions and cannot be pictured. Nothing important is lost. The logic of walking downhill does not care how many directions there are.

Here is the part the tidy version usually leaves out: the walker is in fog. The machine cannot see the landscape – seeing it whole would mean trying every possible setting of every knob, which is more arithmetic than the universe has time for. All it can do is feel the slope directly underfoot: for each knob, would a tiny increase send the loss up, or down, and how steeply? That felt slope has a name – the gradient – and the whole algorithm is: feel the slope, take one small step the downhill way, feel again. Step, feel, step, feel, millions of times.

Walking against the gradient – always in the direction of sharpest drop – is called gradient descent, or in the older phrase, the method of steepest descent. It is where this site takes its name, because this walk is the load-bearing mechanism of the modern field. Not a metaphor for training: the thing itself. When a headline says a model was trained on some vast pile of text, the concrete event being described is this fog-bound walk, taken by a machine, downhill on a loss landscape built from that text.

One decision is left, and it is the walk’s only real knob of its own: how far to move on each step. The field calls this the learning rate, and both of its failure modes are instructive.

Set it too small and nothing goes wrong except your schedule: each step is safe and each step is tiny, and the valley is a week’s shuffle away. Set it too large and something much more interesting happens. From high on one slope you take a stride so long it clears the valley floor entirely and lands you on the far slope – higher than where you started. The ground there is steeper still, so the next stride is even longer. Instead of settling, the walker ricochets between the valley walls, climbing with every bounce, and the loss – the number that was supposed to shrink – grows without limit. The walk has not just failed to find the bottom; it has actively fled it.

The scenario below sets that failure up honestly – a real network, a learning rate deliberately set too hot, and a loss curve you can watch climb instead of fall. It is the most legible failure in machine learning, and having once watched it, you will never confuse training with anything gentle or deliberate. It is a ball bearing bouncing round a bowl, and the step size decides whether it rattles to rest or leaps the rim.

One honest caveat before the primer moves on, because the walk is usually described as if it came with a guarantee.

Nothing about feel-step-feel promises the lowest point. A walker in fog follows the slope into whatever dip it leads to; if a deeper valley lies over the next ridge, the fog hides it, and every direction out of the current dip is uphill, so the walk stops. What gradient descent finds is a low point, not the low point.

In practice – and this is one of the field’s genuine surprises – the low points it finds for large models turn out, again and again, to be good enough, and different runs from different random starts land in different valleys of remarkably similar depth. Why the landscapes of real models are this forgiving is only partly understood; there are persuasive partial explanations involving how many escape routes exist when there are billions of directions to move in, but no tidy settled account. The primer will not dress that up. The procedure works, its results are measured rather than assumed, and the deepest reasons it works this well are still being argued about. You now know exactly as much as the confident version of this paragraph would have told you, plus the truth.