#inference

5 inference interview questions, asked at Amazon, Applied Intuition, Nuro and 2 more companies. Last asked Oct 2026.

Speculative edits: decode a code edit using the original file as the draftCodingLockedA model rewrites a source file after an edit instruction, and most of its output is the original file copied unchanged. Instead of a small draft model, use the original code…
Last asked Oct 2026
Decoder-only LLM architecture and how vLLM serves itAI/ML designLockedTwo parts, asked conversationally at a whiteboard. First explain how a modern decoder-only transformer turns a prompt into a probability distribution over the next token.
Last asked Sep 2026
GPU and inference performance fundamentalsAI/ML designLockedA conceptual round on why GPU code is fast or slow, and where large-model inference spends its time.
Last asked Aug 2026
Design a model inference pipelineAI/ML designDesign a production inference pipeline for a trained model that serves predictions to downstream applications. Treat it as ML infrastructure: how online requests come in and reach…
Last asked Jan 2026
Add a KV cache to GPT inferenceAI/ML designYou have a decoder-only Transformer (a small GPT) that generates text autoregressively by re-running the full sequence for every new token. Design and implement key–value caching…
Last asked Sep 2025