← Questions

Shard an LM head and combine the logits

0
MicrosoftML designLast asked Jan 2026

The output layer maps a hidden state through an [H, V] weight matrix to vocabulary logits. Compare splitting the weight by vocabulary with splitting it by hidden dimension: what each device computes, which reductions are needed, and whether full logits must be gathered before top-k selection or sampling.

Your canvas
Draw your answer hereBoxes, arrows and notes, saved for you to come back to.Sign in to start drawing