Learning from examples

Suppose you have to make a machine that can tell cats from everything else. The obvious way is to write down what a cat is. Whiskers. Pointed ears. Four legs. A tail. Then you meet a cat curled asleep with no legs visible, a Manx with no tail, a cat mid-leap looking like a furry crescent, a photograph of a cat taken from directly above. Every rule sprouts exceptions; every exception needs a rule of its own; and the new rules contradict the old ones in ways that need meta-rules to referee. This is roughly how AI was attempted for its first few decades, by serious people working hard, and it kept failing in the same place: the world contains more unusual cats than any rulebook can hold.

The second way is to stop describing and start showing. Collect a hundred thousand photographs, each labelled cat or not cat, and show them to a machine that adjusts itself a little after every mistake. No definitions, no rules, no expert in the loop. The machine derives its own regularities – whatever combination of texture, shape and context actually separates the cat photographs from the rest.

The point is not that the second way is cleverer. The point is what kind of rules it produces. They are rules nobody could have written down – regularities that do not fit into words at all – and, just as important, rules nobody can easily read back out afterwards. They exist only as the settings of millions of knobs. You trade legibility for capability. That trade is the founding bargain of modern AI, it is the best one-line summary of the field’s history, and chapter 6 collects the bill it runs up.

Here, in full, is the training loop that chapter 0 promised. Show the model one example. It makes a prediction – at the start, a worthless one, because its knobs begin at random. Compare the prediction with the label the example carries. Nudge every knob a tiny amount in whichever direction would have made the prediction slightly less wrong. Show it the next example. Repeat, millions upon millions of times.

That is the whole thing. It deserves a slow second read, because everything impressive in the field is this loop at scale.

Notice how unremarkable each pass is. One example, one comparison, one set of tiny nudges – nothing that looks like insight, nothing worth watching. If you stood over the machine and observed a single adjustment you would learn nothing at all. The behaviour that eventually looks like learning is not in any step; it is in the accumulation of a number of steps no person could sit through.

Hold on to that when you meet the word emergence, which gets used as if something mysterious is added along the way. Between step one and step one billion, nothing is added. It is the same blind nudge, repeated. Each step is understood exactly; what nobody can yet do is read the shape the steps accumulate into – which is a real gap, and chapter 6 is honest about it. But a gap in our reading is not magic in the machine.

There is an asymmetry buried in the loop, and it is worth dragging into the light because it explains so much of what comes later.

The machine is never told what a cat is. No definition ever enters it. The only information it ever receives is whether – here is an example, here is whether you got it right. Everything it accumulates is of the form “settings that predict well”, never “facts about cats”. A model can become excellent at spotting cats while containing nothing anyone could point to and call a concept of a cat. When a later chapter says the model has no beliefs to betray, this is where that starts.

The asymmetry also sharpens a question the loop has been quietly assuming. “Nudge every knob so the prediction is less wrong” only makes sense if wrongness comes in degrees – if a prediction can be a little wrong or badly wrong, and if there is a direction called less. Which way is less wrong? To answer that, wrongness has to stop being a judgement and become a number. That number is the hinge the entire rest of this primer turns on, and it is the next chapter.