Same word, different ids | Steepest Descent

Steepest Descent

prompt

warm-mat same-word-four-ids one-word-many-pieces numbers-come-apart repetition open-door

v1 · gpt-2 small · precomputed prompts

your text · 0 tokens

tap a token to focus every view on it

views

1 · tokens

37 characters

Every spelling gets its own id – or shatters: "HELLO" is rare enough that it arrives as "HELL" + "O". Nothing in the ids says these are the same word.

2 · embedding

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

3 · attention · l3 h6

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

4 · next token

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

1 · tokens

0 tokens · 37 characters

Every spelling gets its own id – or shatters: "HELLO" is rare enough that it arrives as "HELL" + "O". Nothing in the ids says these are the same word.

2 · embedding neighbourhood

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

3 · attention · layer 3, head 6

causal mask on

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

4 · next token

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.