← Questions
Adam, AdamW and decoupled weight decay
0
I was asked this
No reports yet. Sign in to say where you were asked it.
Write Adam's update: the first and second moment estimates, bias correction and the step. Why is adding an L2 penalty to the loss not the same as weight decay once Adam's per-parameter scaling is involved, and what does AdamW change? Which parameters usually get no weight decay in a transformer, and how does the learning-rate schedule interact with the decay?
No official answer yet. Draw yours on the right; publish it and it shows under Community.
Nothing published yet. Publish your canvas and it shows here for others to open and remix.
Draw your answer hereBoxes, arrows and notes, saved for you to come back to.Sign in to start drawing
