decision boundary
epoch 0 · train – · test –
epoch 0 | train –
● A ● B ○ –what to noticewhat to look for
Watch the two curves part company: train keeps falling, test turns and rises. That turn is memorisation starting - the network has enough knobs to store the noise, and nothing inside it can tell. Shrink the network or raise L2 and watch the gap close at the cost of a worse train score. That trade is the whole subject.
instruments
loss
log scale– train –-- test –
gradient magnitude by layer
Gradients shrink as they travel back – the products-of-slopes fact from Maths of Learning ch3, live.