← Questions
RMSNorm versus LayerNorm in a transformer block
0
I was asked this
No reports yet. Sign in to say where you were asked it.
Write the formulas for LayerNorm and RMSNorm and say what each normalizes. What does RMSNorm give up by skipping the mean, why does it work about as well in practice, and why do most recent LLMs use it? Where does the norm sit in a pre-norm versus post-norm block, and how does that choice affect training stability at depth?
No official answer yet. Draw yours on the right; publish it and it shows under Community.
Nothing published yet. Publish your canvas and it shows here for others to open and remix.
Draw your answer hereBoxes, arrows and notes, saved for you to come back to.Sign in to start drawing
