decision boundary
epoch 0 · train – · test –
epoch 0 | train –
● A ● B ○ –what to noticewhat to look for
Three walkers, identical start, identical hill - only the stepping temperament differs. SGD is honest and slow, momentum coasts through the flats, Adam adapts per knob and usually leads early. Notice what does NOT differ: they are all stepping against the same gradient.
instruments
loss
log scale– sgd ·– momentum – adam
gradient magnitude by layer
Gradients shrink as they travel back – the products-of-slopes fact from Maths of Learning ch3, live.