What it can't do

Confident wrongness first, because it is the failure everyone has now met. A model states a fact that is not one – a case that was never tried, a book the author never wrote – fluently, specifically, and with perfect composure. The industry word is hallucination. The word suggests a malfunction, and the suggestion is wrong in an instructive way.

Look back at the loop from chapter 5. The scoresheet scores pieces by how well they fit as text – how strongly the mountain of training text says this-follows-that. Nowhere in the loop is there a step that checks anything against the world. There is no world in the machine; there are knobs, shaped by text. Most text written by people is broadly true, so fitting- the-text and being-true usually travel together – which is exactly what makes the cases where they part company so treacherous. A citation in correct legal format, to a case that does not exist, is high-scoring text: every piece follows beautifully from the last. The machine is not lying; lying needs beliefs, and chapter 1 showed it accumulates settings-that-predict, never beliefs. It is not broken, either. It is doing precisely what it was built to do, on a question where doing-that is not what you needed. Structural, not a bug – the fix cannot be “train it harder”, because the loop being run is the loop working perfectly.

The second failure was promised in chapter 1: the bill attached to learning from examples.

The training loop rewards one thing – low loss on the examples. There are two ways to earn that reward. The intended one: discover the regularities, the shape of the problem, the thing that also works on cats it has never seen. The unintended one: memorise the pile. A model with enough knobs can effectively store the answers, score perfectly on every example it was trained on, and have learned nothing that survives contact with a new one. The field’s word is overfitting, and the machine cannot tell the difference from the inside – both roads make its number small.

The tell is one of the most useful pictures in the field, and it needs two curves. Hold some examples back; never train on them. Watch the loss on the training pile fall, and keep falling. Watch the loss on the held-back examples fall alongside it – then slow, turn, and start to rise. That turn is the moment learning became memorising: from here on, every improvement on the training pile is being bought with performance on reality. The scenario below runs it live – a network with far too many knobs, noisy data worth memorising, and the two curves parting company in front of you. Of everything this course links out to, it is the one most worth sitting with.

The defences – and the whole discipline of when a model can be trusted to generalise – are the business of the Machine Learning course upstairs. For the primer, the shape of the failure is the takeaway: a perfect score on what it studied, and no way to tell, from the score alone, whether anything was learned.

The primer closes where honesty requires it to.

Nobody can read a trained model. Not its users, and not the people who built it. The knobs are all right there – every one inspectable to as many decimal places as you like – and they are billions of numbers, each meaningless alone, none of them corresponding to a fact or a rule or a memory anyone can point to. Why did it answer Paris? The honest answer is: because the arithmetic, with these settings, produced Paris – which explains everything and illuminates nothing. The young field trying to do better, called interpretability, is real science making real progress on small corners: specific little circuits inside models have been traced and understood. Reading off, in general, why a model did what it did is nowhere in sight. When chapter 1 called the founding bargain a trade of legibility for capability, this is the invoice.

Notice how much that one gap explains about the world you actually live in. It is why failures are found by trying things rather than by inspection, why a model can pass a thousand tests and fail strangely on the thousand-and-first, why nobody can quite promise what a system will do before it is deployed. These are not scandals; they are what using an unreadable artefact looks like.

Two endings are on offer at this point, and this primer declines both. One says: just autocomplete, nothing to see. You have seen the structure real prediction demands; “just” was doing dishonest work. The other says: nobody understands anything, be afraid. You have now understood, without one equation, every load-bearing mechanism in the machine – examples, loss, descent, units, tokens, prediction. What remains unknown is real and worth naming precisely, and precisely-named is the opposite of alarming. You hold the same working picture the field holds, minus its mathematics. The mathematics is next door, and it was built for readers exactly one chapter braver than this one assumed.