LLM training concepts screen: RLHF vs DPO, KL, reward hacking, perplexity

A machine learning phone screen made of concept questions on training and post-training large language models, asked one after another. Each takes a few minutes: define the idea precisely, write the formula where there is one, then show you have used it by saying how it behaves in practice and what goes wrong.

Part 1: RLHF with a reward model versus DPO

Compare two ways of aligning a language model with human preferences, starting from a supervised fine-tuned (SFT) model: the classic RLHF pipeline with a reward model and PPO, and direct preference optimization (DPO). Explain the data each one needs, its training stages, its objective, and the role of the reference policy in each. Then compare cost, stability and failure modes and say when you would choose each.

A strong answer covers:

• The RLHF objective, max E[r(x, y)] - beta * KL(pi || pi_ref), and the four models PPO keeps (policy, reference, reward model, value model).
• The DPO loss, -log sigmoid(beta * [log pi(y_w|x)/pi_ref(y_w|x) - log pi(y_l|x)/pi_ref(y_l|x)]), and why it needs no reward model and no sampling: the optimal policy of the KL-regularized problem defines an implicit reward.
• Cost and stability: DPO is a supervised loss over two models, simpler and cheaper; PPO is sensitive to hyperparameters but learns from fresh samples.
• Failure modes: reward hacking and over-optimization against a learned reward in RLHF; DPO overfitting the offline pairs, lowering the probability of both responses, and losing signal as the policy drifts from the data.

Part 2: KL divergence

Define Kullback-Leibler (KL) divergence and explain what it measures. Then explain where it is used with large language models and why, for example as a penalty that keeps a fine-tuned policy close to its reference model, or as a distillation objective.

A strong answer covers:

• Which distribution supplies the samples and which log ratio is averaged, written out.
• Direction: forward KL KL(P || Q) is mass-covering, and reverse KL KL(Q || P) is mode-seeking. Say what each does when one distribution puts zero mass where the other does not (infinite KL, support mismatch).
• In RLHF and GRPO: the penalty β · KL(π_θ || π_ref) keeps the policy near the reference to slow reward hacking and keep fluency; the trade-off in choosing β, and adaptive control.
• In distillation: forward KL to the teacher's distribution, or reverse KL on student samples.

Part 3: reward hacking

Explain reward hacking when a language model is post-trained with reinforcement learning against a reward model or an automatic checker: what it looks like in practice, how you would detect it independently of the reward being gamed, and what you would change in the reward, the data or the training setup to prevent it and to recover when it happens.

A strong answer covers:

• The mechanism: the policy maximizes the proxy, not the intent, and the gap grows with optimization pressure and with distance from the reward model's training data.
• Detection: held-out human or stronger-model evaluation, reward rising while independent quality is flat or falling, tracking length and style statistics, KL from the reference model, reading samples, and adversarial probes of the grader.
• Prevention: a KL penalty to the reference policy, reward-model ensembles and uncertainty penalties, length normalization, rewards that are harder to game (hidden tests, sandboxed execution), refreshing the reward model with on-policy samples, and early stopping on an independent metric.
• Recovery: roll back to an earlier checkpoint, patch the grader, relabel data, and add the exploit to regression tests.

Part 4: importance sampling

Explain importance sampling: what it is and why it works. Then explain where it shows up in training large language models and why it is used there. Go beyond the definition to how it behaves in practice.

A strong answer covers:

• The derivation in one line and the support requirement.
• Variance: weights can explode when q is far from p; the variance grows with the mismatch, and sequence-level ratios multiply token ratios, so they blow up with length.
• In PPO and GRPO: rollouts come from the previous policy, the update uses the ratio to correct for it, and clipping the ratio bounds the update; per-token ratios trade bias for variance.
• Diagnostics: effective sample size, the weight distribution, the fraction of clipped tokens, and KL between the policies.

Part 5: perplexity

Define perplexity for an autoregressive language model, derive how it relates to average negative log-likelihood (cross-entropy), and explain how to read it: what a lower or higher number means, how it is used with today's large language models, and where it stops being a good measure.

A strong answer covers:

• PPL = exp(−(1/N) Σ log p(x_t | x_<t)), a perplexity of k meaning the model is as uncertain as a uniform choice among k tokens.
• Correct evaluation: masking, averaging over target tokens rather than over sequences, and the effect of context length and sliding windows.
• Tokenizer dependence: different vocabularies give incomparable numbers; normalizing to bits per byte or per character fixes the comparison.
• Uses: held-out fit, training and data-quality diagnostics, scaling-law curves, and contamination checks.

Part 6: pipeline parallelism in GRPO training

Explain pipeline parallelism, and how it applies when a language model is trained with GRPO (group relative policy optimization): how the model is split into stages across devices, how micro-batches keep the stages busy, where the idle time comes from, and how the rollout (generation) and training phases of GRPO each fit the pipeline.

A strong answer covers:

• The bubble: with p stages and m micro-batches, roughly (p − 1) / (m + p − 1) of the time is idle in a simple fill-and-drain schedule; more micro-batches or an interleaved 1F1B schedule reduce it, at a cost in activation memory.
• Training phase: a group's samples fit naturally into micro-batches, and advantages are computed per group before the backward pass.
• Generation phase: autoregressive decoding is a poor fit for a deep pipeline, since each token must pass every stage, so latency adds up. Rollouts often run on an inference engine with a different layout, and weights are synced after each update.
• Policy versions and staleness: overlapping generation with training uses rollouts from a slightly older policy, which the importance ratio and clipping correct for within limits.

Follow-ups

• What does beta control in DPO, and what goes wrong when it is too small or too large?
• What changes when reverse KL is used instead of forward KL?
• Reward goes up, but human raters say answers got worse. Walk through how you find the cause.
• Why do PPO-style methods clip the ratio instead of using it directly?
• Why can lower perplexity fail to improve a user's task?
• Rollout lengths vary widely within a group. How does that hurt the pipeline, and what do you do about it?