← Questions

Implement multi-head self-attention

0
AppleMetaUberML designLast asked Sep 2026

Write multi-head self-attention from scratch: project the input to queries, keys and values, split them into heads, compute scaled dot-product attention, combine the heads and apply the output projection. Get the shapes and transposes right, and add a causal mask so a position cannot attend to later positions. Be ready to explain the scaling factor and the cost in sequence length.

Your canvas
Draw your answer hereBoxes, arrows and notes, saved for you to come back to.Sign in to start drawing