#distributed training

7 distributed training interview questions, asked at Meta, Amazon, Google and 4 more companies. Last asked Sep 2026.

+1
Tensor parallelism versus pipeline parallelismML designA transformer is too big for one accelerator. Explain how tensor parallelism and pipeline parallelism each split it: what is cut, what crosses the wire and when, where the idle…
Last asked Sep 2026
Pipeline parallelism in GRPO trainingML designExplain pipeline parallelism and how it applies when a language model is trained with GRPO: how the model is split into stages across devices, how micro-batches keep the stages…
Last asked Aug 2026
Sharded matrix multiply: forward, backward and planted bugsCodingSplit a matrix multiplication across several devices and explain how the pieces combine. Write the forward pass and the gradient computation for that layout in NumPy or PyTorch…
Last asked Jul 2026
Data-parallel and fully sharded matrix multiply over simulated devicesCodingSimulated devices can only exchange tensors through send and receive calls. Finish a data-parallel matrix multiply, then implement a fully sharded version where each device holds…
Last asked Apr 2026
Design a large-scale MoE pretraining pipelineML designDesign the training setup for pretraining a mixture-of-experts language model across many GPUs. Cover how experts and other layers are parallelized, token routing and load…
Last asked Sep 2025
Design an asynchronous RL post-training systemML designDesign a system for post-training a large language model with reinforcement learning where generation (rollouts) and training run asynchronously. Cover how rollout workers and…
Last asked Sep 2025
Design fault-tolerant training for a frontier model on 16k acceleratorsSystem designA 400B-parameter model trains on about 16,000 accelerators for two months. At that scale hardware fails every few hours and each failure stops the synchronous job. Design the…