#batching

7 batching interview questions, asked at Anthropic, Mistral AI, Baseten and 6 more companies. Last asked Oct 2026.

+1
Batched LLM inference API over GPUsAI system designDesign an inference API that batches requests across GPUs efficiently. The discussion goes deep on a few axes: utilization, scaling under spikes, and separating prompt processing…
Last asked Oct 2026
Send requests in batches of five and retry failures after a delayCodingDeliver a list of requests to an API that accepts at most five requests per call. Some requests in a call can fail; a failed request may be retried only after its own delay has…
Last asked Sep 2026
Durable event log writer shared by thousands of threadsCodingMany threads on one server append small opaque records to a single durable log. Design and partly build the writer: how callers hand off records, how writes are batched and…
Last asked Sep 2026
Inference loop with continuous batching over a mock modelCodingYou get a model function that takes a batch of sequences and returns one next token for each. Write the serving loop: keep a fixed number of batch slots full, retire a sequence…
Last asked Sep 2026
Design a PDF-to-Markdown inference API (sync and async)AI system designBuild a service that turns PDF files into Markdown out of three existing functions: one that splits a PDF into page images (CPU-heavy), an OCR model that reads a page image or a…
Last asked Apr 2026
Batching service around an embedding modelCodingPut a service around an embedding model: many callers send texts at once, the service groups them into batches for the model, returns each caller its vectors, and stops without…
Last asked Oct 2025
+4
Design an LLM inference serving platformAI system designDesign the system that serves several large models to many customers with low latency and high GPU utilization.