One word, many pieces | Steepest Descent

Steepest Descent

prompt

warm-mat same-word-four-ids one-word-many-pieces numbers-come-apart repetition open-door

v1 · gpt-2 small · precomputed prompts

your text · 0 tokens

tap a token to focus every view on it

views

1 · tokens

34 characters

"indefatigable" is too rare to earn its own id, so it arrives in pieces. The model never sees the word – only the pieces, in order.

2 · embedding

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

3 · attention · l2 h2

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

4 · next token

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

1 · tokens

0 tokens · 34 characters

"indefatigable" is too rare to earn its own id, so it arrives in pieces. The model never sees the word – only the pieces, in order.

2 · embedding neighbourhood

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

3 · attention · layer 2, head 2

causal mask on

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.

4 · next token

This view needs the model. In v1 it is precomputed for the preset prompts – pick one from the strip above. The tokeniser stays live for anything you type.