#transformers

19 transformers interview questions, asked at Amazon, Applied Intuition, Google and 12 more companies. Last asked Oct 2026.

+1
Find the planted bugs in a small Transformer training scriptCodingA small Transformer, its training script and some tests are provided, with a few bugs hidden in them. Find and fix them so the loss goes down and the outputs are correct, and…
Last asked Oct 2026
+2
Transformer and attention fundamentalsML designExplain how a transformer works from the inside: how self-attention is computed, why the scores are scaled, how encoder and decoder masking differ, and what multiple heads buy…
Last asked Sep 2026
+2
Implement multi-head self-attentionML designWrite multi-head self-attention from scratch: project the input to queries, keys and values, split them into heads, compute scaled dot-product attention, combine the heads and…
Last asked Sep 2026
RMSNorm versus LayerNorm in a transformer blockML designWrite the formulas for LayerNorm and RMSNorm and say what each normalizes. What does RMSNorm give up by skipping the mean, why does it work about as well in practice, and why do…
Last asked Sep 2026
Review a compact GPT-style transformer implementationML designYou are given a short implementation of a GPT-style decoder-only transformer. Read it and review it: explain what each part does (embeddings, attention with its causal mask…
Last asked Jul 2026
Rapid-fire questions on LLM internalsML designA fast round of short questions on how LLMs work: transformer and attention internals, training and fine-tuning choices, and how models are run in practice. Each answer should be…
Last asked Jul 2026
LLM foundations: architecture, adaptation and steeringML designA broad conversation on large language models: how the architecture works, the ways to adapt a model to a task (full fine-tuning, parameter-efficient methods, prompting), and the…
Last asked Jun 2026
QKV attention programming with einsum fluencyML designA short programming screen around sequence models and scaled dot-product attention, plus quick ML knowledge questions. Reading equations and expressing the math with einsum-style…
Last asked Feb 2026
Implement multi-head attention and LLM samplingCodingImplement multi-head self-attention from scratch, then the sampling step of a language model: given logits, pick the next token with temperature, top-k and top-p (nucleus)…
Last asked Feb 2026
Multi-head attention with a KV cache and grouped-query attentionML designImplement the attention layer of a decoder-only transformer from scratch in three steps that build on each other: 1. Standard multi-head attention with a causal mask. 2. A…
Last asked Jan 2026
Transformers and the challenges of deploying LLMsML designExplain how a Transformer works, then discuss recent advances in large language models, how you would apply one in the team's product, and the main problems with deploying LLMs in…
Last asked Dec 2025
Implement a transformer block with SwiGLUCodingWrite a transformer block from scratch: multi-head self-attention, the residual connections and normalisation, and a feed-forward layer that uses the SwiGLU gate (two parallel…
Last asked Dec 2025
Implement a decoder-only GPT modelCodingWrite a small decoder-only transformer from scratch as four classes: multi-head self-attention with a causal mask, the feed-forward block, a decoder layer that combines them with…
Last asked Nov 2025
Debug a vision transformer notebookCodingA notebook trains a small vision transformer on handwritten digits but performs badly. Find and fix the implementation bugs, paying attention to details like masking and padding…
Last asked Oct 2025
RNNs, LSTMs, Transformers and model predictive controlML designA technical conversation on the sequence models in your projects: how RNNs, LSTMs and Transformers differ, why LSTMs help with long sequences, what attention buys you, and when…
Last asked Sep 2025
Add a KV cache to GPT inferenceML designYou have a small GPT model with plain autoregressive generation that recomputes attention over the whole sequence for every new token. Add a key-value cache so each step only…
Last asked Sep 2025
Predict the reaction factor of two moleculesML designA dataset gives pairs of molecules and the reaction factor measured for each pair. Build a model to predict the reaction factor for new pairs. Walk through exploring the data…
Last asked Sep 2025
Scaled dot-product attention from scratchCodingWrite scaled dot-product attention by hand: from queries, keys and values, compute the scores, scale them by the square root of the key size, apply an optional mask, take a…
Last asked Sep 2025
Attention and a small Transformer with a hand-written backward passCodingImplement scaled dot-product attention and a small autoregressive Transformer block from scratch, including the causal mask. Then write the backward pass by hand (no autograd) for…
Last asked Aug 2025