#model serving

7 model serving interview questions, asked at Amazon, Anthropic, Character.AI and 4 more companies. Last asked Sep 2026.

Design a cache for large model weights on GPU nodesSystem designContainers serving a model download its weights (around 40 GB) every time they start cold. Design a cache so that the second start is fast. A model version never changes once…
Last asked Sep 2026
Scale and observe a production ML serving serviceSystem designAn existing service takes prediction requests, does asynchronous preprocessing and feature lookups, then calls a versioned model. Traffic is bursty, inference is expensive, and…
Last asked Aug 2026
Design a platform to train and serve ML modelsAI/ML designDesign a platform that lets teams train machine learning models and serve them in production. Cover data access, training jobs, storing and versioning models, deployment, and…
Last asked Aug 2026
Design a model inference pipelineAI/ML designBilled as ML design but really ML infrastructure: design a pipeline that serves a trained model's predictions. Cover how requests or data arrive, batching, GPU use, model versions…
Last asked Jan 2026
Design large-scale model inference serving with back-of-envelope sizingSystem designDesign an inference system for a model used by 100 million daily active users. Do the back-of-envelope estimates for memory and bandwidth, then explain how you would reduce…
Last asked Nov 2025
Design a transformer-based TSP route optimizerSystem designCustomers send weighted complete graphs and want a near-optimal traveling-salesperson tour back. The team trains transformer models on generated graphs solved by a classical…
Last asked Nov 2025
Design an image generation productSystem designDesign a system that generates images (and possibly other media) from user prompts. Cover the model pipeline, serving on GPUs with acceptable latency and cost, safety filtering of…
Last asked Aug 2025