How numbers tokenise | Steepest Descent

Steepest Descent

prompt

warm-mat same-word-four-ids one-word-many-pieces numbers-come-apart repetition open-door

v1 · gpt-2 small · precomputed prompts

your text · 0 tokens

tap a token to focus every view on it

views

1 · tokens

26 characters

"1999" was common enough in training text to earn one id; "3.99" arrives as three. The model does arithmetic on pieces like these, which is part of why it is bad at it.

2 · embedding

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

3 · attention · l2 h2

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

4 · next token

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

1 · tokens

0 tokens · 26 characters

"1999" was common enough in training text to earn one id; "3.99" arrives as three. The model does arithmetic on pieces like these, which is part of why it is bad at it.

2 · embedding neighbourhood

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

3 · attention · layer 2, head 2

causal mask on

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

4 · next token

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.