LLM and generative AI system design interview questions
Candidates' reports name 107 LLM and generative AI system design questions at 31 AI companies. The most are at Anthropic (17), OpenAI (14) and Cohere (6). The most asked themes are LLM products and platforms (22), agents and tool use (21) and LLM inference and serving (20), and the newest was asked in Nov 2026. Each question below has the company, the round and when it was last asked.
Updated
| Company | Questions | Last asked |
|---|---|---|
| Anthropic | 17 | Oct 2026 |
| OpenAI | 14 | Oct 2026 |
| Cohere | 6 | Oct 2026 |
| Decagon | 6 | Sep 2026 |
| Harvey | 6 | Oct 2026 |
| Mercor | 6 | Sep 2026 |
| Scale AI | 6 | Oct 2026 |
| Mistral AI | 5 | Sep 2026 |
| Lambda | 4 | Sep 2026 |
| LangChain | 4 | Sep 2026 |
| SpaceXAI | 4 | Feb 2026 |
| Waymo | 4 | Sep 2026 |
| Baseten | 3 | Oct 2026 |
| Hebbia | 3 | Jun 2026 |
| Modal | 3 | Sep 2026 |
| Nuro | 3 | Oct 2026 |
| Perplexity | 3 | Oct 2026 |
| Applied Intuition | 2 | Nov 2026 |
| Cognition | 2 | Sep 2026 |
| Cursor | 2 | — |
| ElevenLabs | 2 | Nov 2026 |
| Glean | 2 | Sep 2026 |
| Google DeepMind | 2 | Nov 2026 |
| Hugging Face | 2 | — |
| Sierra | 2 | Oct 2026 |
| Together AI | 2 | Oct 2026 |
| Abridge | 1 | Aug 2026 |
| Character.AI | 1 | Nov 2025 |
| Fireworks AI | 1 | Aug 2025 |
| Rogo | 1 | Jun 2026 |
| Runway | 1 | Aug 2025 |
LLM products and platforms 22
- Design a real-time AI voice agent · ElevenLabs design review, Decagon technical phone screenNov 2026
- Design a backend that asks one LLM question of 100,000 documents · Harvey ai system designOct 2026
- Design an AI chatbot app with streaming replies and saved chats · OpenAI system design, Baseten system design, Waymo system designOct 2026
- Design an AI prompt playground · Anthropic system design, OpenAI technical phone screenOct 2026
- Design a resumable streaming voice reply for an AI chat · Sierra ai system designOct 2026
- Research engineering training discussion · Harvey project deep diveSep 2026
- Speed up object-backed APIs, then design LLM observability · LangChain ai system designSep 2026
- Research ML fundamentals screen: AdamW, RMSNorm, parallelism, MoE, DPO, diffusion · Mistral AI technical phone screenSep 2026
- Design a streaming transcription service · Abridge system designAug 2026
- Design an AI gateway · Decagon technical phone screenAug 2026
- Review a compact GPT-style transformer implementation · Mercor ai/ml designJul 2026
- Design per-user model usage quotas for a mobile app · OpenAI technical phone screenMay 2026
- Mechanistic interpretability experiment on double descent · Anthropic system designApr 2026
- Debug a GRPO training script and defend the RL math · Anthropic system designMar 2026
- QKV attention programming with einsum fluency · Anthropic technical phone screenFeb 2026
- ML fundamentals deep dive for ML roles · Scale AI ai system designFeb 2026
- Implement multi-head self-attention from scratch · Together AI ai system designFeb 2026
- Design the front end of a desktop AI chat app · Anthropic technical phone screenSep 2025
- Design a batch job orchestrator over a rate-limited LLM API · Scale AI ai system designMay 2025
- Design review: usage billing for an LLM API · Anthropic system design
- Design permission-aware enterprise search · Glean system design
- Dense versus mixture-of-experts at fixed compute · Hugging Face ai system design
Agents and tool use 21
- Design durable execution for long-running agent queries · Perplexity system designOct 2026
- Rule-driven file review agent · Harvey ai system designOct 2026
- Design an enterprise research agent with internal and web sources and citations · Cohere technical phone screenOct 2026
- Build an event-driven engineering automation on a coding agent's API · Cognition system designSep 2026
- Design a RAG question-answering agent over a firm's documents · Harvey ai system designSep 2026
- Design an evaluation platform for a design-editing AI agent · OpenAI system designSep 2026
- Design a platform for long-horizon agent training environments · Harvey ai system designSep 2026
- Autonomous vulnerability remediation for a GitHub project with a coding agent · Cognition system designSep 2026
- Design an online store with an AI shopping agent · ElevenLabs design reviewAug 2026
- Design a customer-service AI agent with tools and guardrails · Sierra ai system design, Decagon ai system designAug 2026
- Build a RAG agent on the Mistral API, then defend its design · Mistral AI system designJul 2026
- Debugging and evaluating LLM agents in production · LangChain ai system designJun 2026
- Build and demo a customer-support agent with LangGraph · LangChain take-homeJun 2026
- Design a multi-agent chat backend · Hebbia system designJun 2026
- Design a multi-agent document search and visualization product · Hebbia system designJun 2026
- Design an agent harness and its evaluation system · OpenAI system designApr 2026
- Design an insurance claims agent with RAG · Scale AI ai system designApr 2026
- Design an agent workflow that produces a feature-length video · SpaceXAI technical phone screenFeb 2026
- Design an evaluation harness for a customer-support AI agent · Decagon ai system design
- Design an AI coding agent sandbox · Cursor, Anthropic
- Design a multi-agent system that redlines a contract against a playbook · Harvey ai system design
LLM inference and serving 20
- Design a simulation job scheduler that matches jobs to hardware · Applied Intuition system designNov 2026
- Design an LLM inference API that batches requests on GPUs · Anthropic system design, Baseten technical phone screenOct 2026
- Design an LLM inference serving platform · OpenAI system design, Lambda recruiter screen, Baseten system designOct 2026
- Design ChatGPT · Mercor system designSep 2026
- Design a cache for large model weights on GPU nodes · Modal ai system designSep 2026
- Post-train a 7B model to code better for one enterprise · Cohere ai system designSep 2026
- Design review: a batched GPU inference API · Anthropic system designSep 2026
- Design a streaming AI feature for a large audience · OpenAI system designAug 2026
- Scale and observe a production ML serving service · Anthropic system designAug 2026
- Design ChatGPT streaming with resumable responses and long conversations · OpenAI system designJul 2026
- Design a PDF-to-Markdown inference API (sync and async) · Mistral AI system designApr 2026
- Design a model inference pipeline · Nuro technical phone screenJan 2026
- Model compression: quantization-aware training, distillation and more · Waymo technical phone screenNov 2025
- Design large-scale model inference serving with back-of-envelope sizing · Waymo technical phone screenNov 2025
- Design a transformer-based TSP route optimizer · Character.AI ai system designNov 2025
- Add a KV cache to GPT inference · Applied Intuition technical phone screenSep 2025
- KV cache eviction for million-token contexts · Hugging Face ai system design
- Design review: rolling out a new model version · Anthropic system design
- Design a multi-tenant fine-tuning and LoRA serving platform · Mistral AI system design
- Apply model edits to a file with an AST and model routing · Cursor ai system design
Training, fine-tuning and GPUs 16
- Schedule video processing across CPU and GPU workers · Nuro system designOct 2026
- Design a GPU scheduling platform · Together AI technical phone screen, Nuro ai system design, Lambda system designOct 2026
- Design a distributed AI model downloader · Anthropic system designOct 2026
- Design a video generation service on a scarce GPU pool · OpenAI system designSep 2026
- Design a warm pool of GPU containers · Modal system designSep 2026
- Design a scheduler for bursts of short serverless tasks · Modal system designSep 2026
- Design a high-throughput object store for ML datasets and checkpoints · Lambda system designSep 2026
- Design a Kubernetes control plane for GPU workloads with topology and health-aware draining · Lambda system designSep 2026
- Make an LLM recognize questions past its training cutoff · Cohere ai system designSep 2026
- Matmul performance modeling on a fixed GPU · Anthropic system designSep 2026
- Rapid-fire questions on LLM internals · Mistral AI system designJul 2026
- Evaluate noisy data for LLM post-training · Mercor technical phone screenMay 2026
- Improve an LLM's reasoning on a domain task · Cohere ai system designDec 2025
- Explain fully sharded data parallel (FSDP) training with a diagram · Fireworks AI system designAug 2025
- Design review: checkpointing a large training run · Anthropic system design
- Design fault-tolerant training for a frontier model on 16k accelerators · Google DeepMind system design
RAG, retrieval and search 14
- Design a pipeline around embedding and classification services · Scale AI ai system designOct 2026
- Design a work-together distance between two employees · Glean system designSep 2026
- LLM application engineering: RAG, context limits, state and prompt injection · Mercor technical phone screenSep 2026
- Design an internal AI assistant for employees · Decagon technical phone screenSep 2026
- ML design for retrieval: embeddings, hybrid search and ranking · OpenAI system designSep 2026
- Manage the context window in an applied LLM system · Rogo ai system designJun 2026
- Design an LLM chat over SEC filings · Hebbia technical phone screenJun 2026
- Mine new information from a huge unlabeled image corpus · OpenAI system designMay 2026
- Design candidate search for a talent marketplace · Mercor ai/ml designMay 2026
- Design a RAG system over company docs · OpenAI system design, SpaceXAI system designMar 2026
- Improve a chatbot's answers to questions it doesn't know · OpenAI system designDec 2025
- Design a feed recommendation system · Perplexity ai system design, OpenAI technical phone screenSep 2025
- Design a video search system · Runway technical phone screenAug 2025
- Design an answer engine with citations · Perplexity ai system design
Evals, safety and guardrails 14
- A model about to ship regressed on safety evals · Google DeepMind codingNov 2026
- Defend your own research paper under hard questions · Cohere ai system designSep 2026
- Design the security architecture for an LLM chatbot · Decagon ai system designSep 2026
- Design simulation infrastructure to evaluate a self-driving model on limited compute · Waymo system designSep 2026
- Demo an LLM observability and evaluation tool to newcomers · LangChain take-homeApr 2026
- Build an eval harness for local models served by Ollama · Mercor technical phone screenMar 2026
- Design guardrails and fallbacks for a reliable LLM service · Anthropic system designFeb 2026
- Prompt design with evaluation scaffolding · Anthropic system designJan 2026
- Explain LLM training, reinforcement learning and evaluation · Cohere ai system designDec 2025
- How would you prevent hallucinations in an LLM-based system? · SpaceXAI recruiter screenNov 2025
- House price prediction from historical sales, end to end · SpaceXAI technical phone screenJul 2025
- Implement a universal adversarial attack on GPT-2 from a paper · Scale AI ai system designJul 2025
- Design LLM-assisted grading of contributor work · Scale AI ai system designMay 2025
- Design review: an eval gate for model launches · Anthropic system design
More from the data