← Questions

Adam, AdamW and decoupled weight decay

0
Mistral AIML designLast asked Sep 2026

Write Adam's update: the first and second moment estimates, bias correction and the step. Why is adding an L2 penalty to the loss not the same as weight decay once Adam's per-parameter scaling is involved, and what does AdamW change? Which parameters usually get no weight decay in a transformer, and how does the learning-rate schedule interact with the decay?

Your canvas
Draw your answer hereBoxes, arrows and notes, saved for you to come back to.Sign in to start drawing