#evaluation

23 evaluation interview questions, asked at Amazon, Apple, Meta and 10 more companies. Last asked Sep 2026.

Design simulation infrastructure to evaluate a self-driving model on limited computeSystem designStart from an open question: how would you evaluate a self-driving model? Then design the infrastructure: running thousands of simulations when there is not enough compute…
Last asked Sep 2026
Design retrieval for a coding agent over a code baseAI system designA coding agent is an LLM assistant that works inside a software code base: it reads code, answers questions about it and makes edits. It depends on retrieval to put the right code…
Last asked Sep 2026
How would you verify a smile detector in a photo app?ML designYou are on the Google Photos team and your feature detects smiling faces in photos. How would you check that the detection is correct? Cover what counts as a smile and who decides…
Last asked Sep 2026
Design a pipeline that distills reusable agent skillsAI system designEnterprise customers run AI agents on their own tasks. Design a pipeline that automatically extracts reusable skills from that usage: a skill is a named package of instructions…
Last asked Sep 2026
What perplexity tells you about a language modelML designDefine perplexity and explain how to read it: what a lower or higher number means, how it relates to cross-entropy, how it is used for today's large language models, and where it…
Last asked Aug 2026
Design an evaluation platform for a design-editing AI agentAI system designAn AI agent edits design files on request. Design the platform that evaluates it: the test cases, how an edited design is judged (automatic checks, model graders, human review)…
Last asked Jul 2026
Walk through an agent evaluation codebase for PDFsCodingYou are shown a codebase that evaluates an agent working over PDF documents. Walk the interviewer through it: how the pieces fit together, how the evaluation runs and is scored…
Last asked Jul 2026
Design an LLM-based coding assistantAI system designDesign an AI coding assistant that helps developers inside their editor: completing code, answering questions about a codebase and making edits on request. Discuss how it gathers…
Last asked Jun 2026
Design an agent that investigates support tickets automaticallyAI system designDesign an AI agent that picks up incoming support or on-call tickets and investigates them before a human does: it reads the ticket, gathers evidence from logs, dashboards, code…
Last asked Apr 2026
Design an agent harness and its evaluation systemAI system designDesign the harness that runs an AI agent (the loop that calls the model, runs its tools and manages its context) together with the system that evaluates it. Cover how the agent…
Last asked Apr 2026
Design a code review agentAI system designDesign an AI agent that reviews pull requests: it reads the diff and the surrounding code, leaves comments on bugs, risky changes and style problems, and may suggest fixes. Cover…
Last asked Mar 2026
Design quality checks for spreadsheet data used to train an LLMML designYou are building training or evaluation data from spreadsheets for a language model. Design the checks that catch bad examples: structural problems, wrong or inconsistent values…
Last asked Mar 2026
Analyze the failures of a vision modelML designA vision model works well on average but fails on some inputs. Explain how you would find and group the failures (slicing by condition, looking at examples, confusion analysis)…
Last asked Mar 2026
Prompt design with evaluation scaffoldingML designIn a notebook, design prompts for a defined task and write the surrounding scaffolding that evaluates them. The exercise tests task framing and measurement more than clever…
Last asked Jan 2026
Design a computer-use agentAI system designDesign an agent that operates a computer the way a person does: it looks at the screen, decides the next action and clicks or types until a task is done. Cover how the model…
Last asked Jan 2026
Make LLM output match a word processor's formattingAI system designA writing assistant inside a word processor generates text that must fit the document's formatting: styles, headings, lists, tables. Design how you get the model to produce…
Last asked Jan 2026
Debug a model that does worse online than offlineML designA model scored well in offline evaluation, but after launch its online metrics are worse. How do you find out why? Walk through the likely causes (training-serving skew in…
Last asked Jan 2026
Train and improve a binary classifier with scikit-learnCodingGiven a small labelled dataset, train a binary classifier with scikit-learn, report a sensible evaluation metric, and then improve it through feature handling, model choice or…
Last asked Dec 2025
Design an LLM output quality validation systemAI system designDesign a system that checks the quality of a language model's answers before and after they reach users. Cover what "quality" means for the product, automatic checks and…
Last asked Dec 2025
How would you prevent hallucinations in an LLM-based system?ML designA screening question: how do you keep an LLM-based system from making things up? Cover grounding the model with retrieval and citations, constraining outputs, checking answers…
Last asked Nov 2025
Design a RAG system end to endAI system designDesign a retrieval-augmented generation system from documents to answers: ingesting and chunking documents, embeddings and the index, retrieval and reranking, the prompt to the…
Last asked Sep 2025
Annotator agreement and LLM judges versus human judgesML designExplain annotator agreement rates (such as Cohen's kappa) and their limits. Compare human raters with an LLM as a judge: cost, consistency, bias and where each fails. Then say how…
Last asked Aug 2025
House price predictionML designHow would you build a model that predicts house prices? Cover the features, handling missing values and outliers, model choice (from linear regression to gradient-boosted trees)…
Last asked Jul 2025