Entropy and cross-entropy

Chapter 5 ended on the real question of training: the model holds a spread of belief, the world then reveals what actually happened, and someone must say – with a number – how badly the belief performed. The number the field chose is built from one psychological observation made precise: being wrong is being surprised.

If the model gave the true outcome a 95% share, what happened was expected; mild surprise; small penalty. A 3% share: badly surprised; large penalty. A 0% share deserves special attention – the model called the truth impossible – and the ruler this chapter builds hands that case an unbounded penalty, which is not a quirk but a design decision with teeth: a model must never be completely certain and wrong.

One number per revelation, big when belief failed the world. Average it over everything the model sees, and the primer’s “wrongness” – the number the foggy walker was descending all along – finally has its construction.

Before grading beliefs against the world, one detour that earns its keep: grade a spread of belief against itself. Ask: living under this belief, how surprised do I expect to be, on average? Chapter 5 built the exact instrument for such questions – the expectation.

A fair coin: every toss surprises you a little, and equally. A heavily loaded coin: mostly no surprise, rarely a jolt; on average, very little. A die is more surprising on average than a coin – more ways for the world to jump. This expected self-surprise is called entropy, and it is uncertainty itself, measured in bits: how much genuine unpredictability a situation holds, before anyone makes any mistakes about it.

It sets a floor. Even a perfect model of a coin toss is surprised by coin tosses – the world contains real randomness, and no belief can be less surprised than the world is surprising. Keep the floor in mind; it is what the next section’s number sits on top of.

Now the finale. The world runs on one spread – what actually tends to happen. The model runs on another – what it believes. Living by the model’s map while the world deals the cards means: surprises are sized by the model’s beliefs, but arrive at the world’s frequencies. The average surprise of that mismatched life is called cross-entropy, and it is the number.

Two facts give it its shape. It can never beat the floor: navigating by the wrong map can never make the world less surprising than it intrinsically is. And the excess above the floor is a pure tax on wrongness – it measures nothing about the world and everything about the map’s errors. Shrinking the number therefore has only one lever: fix the map. Which is why it works as a loss.

Stand back and watch the six pieces click. A token becomes a vector (chapter 1) so that similarity is arithmetic. Layers map and bend those vectors (chapter 2). The machine’s raw scores become a spread of belief (chapter 5), and the spread is graded by its surprise at what the world actually did (this chapter). That grade is a smooth function of every knob, so the chain rule can feel it (chapter 3), gather the feelings into one arrow, and step against it (chapter 4). Loop until the surprise stops falling.

That paragraph is the entire training story of modern machine learning. Nothing in the later courses replaces it; they fill in what the layers compute, what the data demands, and what the trained result can and cannot be trusted to do. The mathematics you were promised would be the shortcut has now paid out in full – and the next course starts using it.