← Questions

Design an LLM inference API that batches requests on GPUs

0
AnthropicBasetenSystem designLast asked Oct 2026

An AI system design interview question asked in Anthropic's system design round and at Baseten, last asked Oct 2026. It tests llm serving, batching and gpu.

Design an API that serves a large language model to many clients and batches their requests on GPUs to use the hardware well. Each client sends a prompt and waits for (or streams) the response.

Sign in to unlock this question

The full prompt, the worked answer and your own canvas or code with the AI tutor. New accounts get 5 credits: each one unlocks a question for good.

Where it's asked

  • Anthropic · System design · Last asked Oct 202655 min · Design discussion, often typed into a shared Google Doc rather than drawn; some rounds let you share any drawing tool · No AI
  • Baseten · Technical phone screen · Last asked Jul 202660 min · Live in CoderPad: a few minutes of intro, then 45 to 50 minutes on one practical problem

What it tests

llm serving · batching · gpu

Your canvas and the AI tutor open when you unlock this question.