#kv cache

5 kv cache interview questions, asked at Amazon, Anthropic, Applied Intuition and 4 more companies. Last asked Sep 2026.

Decoder-only LLM architecture and how vLLM serves itML designExplain how a decoder-only language model is built, from tokens through the stack of attention and feed-forward layers to predicting the next token. Then explain how an inference…
Last asked Sep 2026
Multi-head attention with a KV cache and grouped-query attentionML designImplement the attention layer of a decoder-only transformer from scratch in three steps that build on each other: 1. Standard multi-head attention with a causal mask. 2. A…
Last asked Jan 2026
Add a KV cache to GPT inferenceML designYou have a small GPT model with plain autoregressive generation that recomputes attention over the whole sequence for every new token. Add a key-value cache so each step only…
Last asked Sep 2025
KV cache eviction for million-token contextsAI system designDesign an eviction policy for serving very long contexts on limited GPU memory, comparing sliding windows, attention sinks, heavy-hitter eviction, quantization and tiered or…
+4
Design an LLM inference serving platformAI system designDesign the system that serves several large models to many customers with low latency and high GPU utilization.