#llm serving

12 llm serving interview questions, asked at Anthropic, OpenAI, Mistral AI and 7 more companies. Last asked Oct 2026.

+1
Batched LLM inference API over GPUsAI system designDesign an inference API that batches requests across GPUs efficiently. The discussion goes deep on a few axes: utilization, scaling under spikes, and separating prompt processing…
Last asked Oct 2026
Post-train a 7B model to code better for one enterpriseML designAn enterprise customer wants a 7-billion-parameter model that writes better code for them. Design the post-training pipeline end to end: collecting and cleaning data, supervised…
Last asked Sep 2026
Inference loop with continuous batching over a mock modelCodingYou get a model function that takes a batch of sequences and returns one next token for each. Write the serving loop: keep a fixed number of batch slots full, retire a sequence…
Last asked Sep 2026
Design a streaming AI feature for a large audienceAI system designA consumer product wants to add an AI feature whose answers stream to millions of users in real time. Design the path from the client to the model: how responses stream, how…
Last asked Aug 2026
Design ChatGPT streaming with resumable responses and long conversationsAI system designYou have already sketched a ChatGPT-style chat product. Now go deeper on three follow-ups: how replies stream to the browser (SSE or something else, and why), how a reply survives…
Last asked Jul 2026
Design a PDF-to-Markdown inference API (sync and async)AI system designBuild a service that turns PDF files into Markdown out of three existing functions: one that splits a PDF into page images (CPU-heavy), an OCR model that reads a page image or a…
Last asked Apr 2026
Fast expensive tokens versus slow cheap tokensAI system designAn LLM provider offers fast but expensive tokens and slow but cheap ones. How would you decide which to use for a product? Discuss which features need low latency, batch versus…
Last asked Jan 2026
Batching service around an embedding modelCodingPut a service around an embedding model: many callers send texts at once, the service groups them into batches for the model, returns each caller its vectors, and stops without…
Last asked Oct 2025
+4
Design an LLM inference serving platformAI system designDesign the system that serves several large models to many customers with low latency and high GPU utilization.
Design a multi-tenant fine-tuning and LoRA serving platformAI system designA model provider lets customers fine-tune its open-weight models on their own data through an API, then call the fine-tuned models through the same chat endpoint. Most use LoRA…
Design review: rolling out a new model versionAI system designYou are reviewing a launch plan written by the serving team. Critique it: rank the risks, propose changes, and say what must be true before you would approve it. You have 45…
+3
Design ChatGPTAI system designDesign a consumer chat product backed by a large language model: conversations, streamed replies, history across devices, at 100M weekly users.