Words become numbers

The machine assembled over the last three chapters has one diet. It multiplies numbers by knobs and adds the results. It never does anything else, and nothing in the coming chapters changes that. So before it can touch language, language has to become numbers – and the bridge the field built has two halves, one blunt and one strange.

The blunt half first: chop the text into pieces. The pieces are called tokens, and the thing to unlearn immediately is that they are words. Common words do tend to survive as single pieces – the, and, cat. Rarer words get broken into reusable fragments, so unbelievable might travel as something like un, believ, able. Punctuation marks are pieces; spaces are usually carried on the front of pieces; and where the boundaries fall looks arbitrary until you hear how they were chosen: by counting. Take a mountain of text, find the chunks of characters that recur most often, and give the commonest few tens of thousands of chunks a piece each. No grammar was consulted, no dictionary. Frequency did all of it.

The fastest cure for thinking the machine reads the way you do is to watch your own name go through the chopper. A common first name is frequent enough in text that it typically arrives whole, as one piece. An uncommon surname is not, so it shatters into fragments – chosen, remember, by frequency, not by sense, so the break points land mid-syllable, indifferent to anything your name means to you. To the machine your name is not a name; it is two or three pieces that happen to be common in other people’s words.

This one fact quietly explains a family of famous oddities. Ask a language model to count the letters in a word, or to work with rhymes and anagrams, and it is often strangely poor at it – because it never sees letters. The letters were gone before the model’s first multiply. It sees pieces, and what a piece is made of is simply not part of what it is given. The mental picture to keep is a strip of text with scissor-cuts in odd places – some between words, some through them, none placed by anything that reads.

Now the strange half of the bridge. Each piece in the vocabulary is given a position in a space. Not our space – a space with hundreds or thousands of directions, which no one can picture and no one needs to: a position there is just a long list of coordinates, one number per direction. The position is called an embedding, and here is the part that matters: nobody chooses the positions. The coordinates are knobs, exactly like every other knob in the model, nudged by the same guess-check-adjust walk as everything else.

So the positions end up wherever prediction pushed them. And what prediction discovers, over billions of examples, is that pieces used in similar surroundings are cheapest to treat similarly – so Tuesday drifts near Wednesday, cold near chilly, and the fragments of your surname end up wherever their statistics put them. The neighbourhoods that result can look uncannily like a map of meaning. Nobody drew that map. It is a fossil of the prediction task: the shape the knobs settled into because that shape made the loss smallest.

This is the paragraph this chapter exists for, so it will not be rushed.

An embedding records which company a piece keeps – a compressed summary of the contexts it appeared in, across a mountain of text. That is all the training signal ever contained, so that is all the position can hold. Keeping-company is often an excellent stand-in for meaning; words used identically usually do mean similar things, which is why the map looks the way it does. But a record of contexts is not a grasp of content, and the two come apart exactly where you would expect: a model can place a word perfectly and still misuse the thing the word is about.

You may have met this chapter’s most famous party trick: take the position of king, subtract the position of man, add woman, and land near queen – meaning, apparently, as arithmetic. The trick is real and worth seeing once. It is also worth the caveat in the same breath: it works for a handful of well-chosen pairs and much less reliably in general, and for every clean example there are many where the same arithmetic lands nowhere useful. It demonstrates that the space has real, learned structure. It does not demonstrate an algebra of meaning, and it is routinely presented as one.

Why does keeping-company carry as far as it evidently does? That is a genuinely open question – one of the honest surprises of the field, not a mystery invented for drama. The primer’s working rule, here and everywhere: describe what the numbers do, and decline the free upgrade to what they mean. With text now numbers, the machine can finally be put to work on language – and the next chapter assembles everything into the loop you actually came to understand.