#parallelism

7 parallelism interview questions, asked at Google, Mistral AI, Amazon and 3 more companies. Last asked Sep 2026.

Mixture of experts and expert parallelismML designExplain how a mixture-of-experts transformer layer works and follow a batch of tokens through it: the router, top-k selection, the all-to-all exchange that sends tokens to experts…
Last asked Sep 2026
+1
Tensor parallelism versus pipeline parallelismML designA transformer is too big for one accelerator. Explain how tensor parallelism and pipeline parallelism each split it: what is cut, what crosses the wire and when, where the idle…
Last asked Sep 2026
Pipeline parallelism in GRPO trainingML designExplain pipeline parallelism and how it applies when a language model is trained with GRPO: how the model is split into stages across devices, how micro-batches keep the stages…
Last asked Aug 2026
Count distinct strings with your own hash table, then on a GPUCodingGiven an array of strings, return how many distinct strings it contains, without using library hash sets or maps: build your own hash table with a hash function and collision…
Last asked Jul 2026
Data-parallel and fully sharded matrix multiply over simulated devicesCodingSimulated devices can only exchange tensors through send and receive calls. Finish a data-parallel matrix multiply, then implement a fully sharded version where each device holds…
Last asked Apr 2026
Multiply two matrices across two GPUsML designMatrices A and B must be multiplied using two GPUs, with both matrices split between them. How do you partition the work (by rows, by columns, by blocks), what has to move between…
Last asked Sep 2025
Dense versus mixture-of-experts at fixed computeML designWith a fixed activated-parameter budget, compare a dense model against a mixture-of-experts on training cost, memory, communication, stability, inference and engineering effort…