Mechanistic interpretability experiment on double descent

Design and run a small-model experiment reproducing sample-aspect double descent, explain the phenomenon from theory, and propose a remedy. The deliverable is the experiment plus the explanation.

A small, fast setting is fine: linear regression on Gaussian features, fitted with minimum-norm least squares.

import numpy as np


def make_task(p=50, seed=0): ...
def sample(rng, n, w_true, noise): ...        # -> x, y, eps
def fit(x, y, ridge=0.0): ...
def sweep(ns, p=50, noise=0.5, ridge=0.0, trials=20, seed=0):
    """Rows of (n, test MSE, ||w||, ||noise part of w||, smallest singular value of X)."""
def theory_mse(n, p, noise, w_norm_sq=1.0): ...

Mechanistic interpretability experiment on double descent

I was asked this

Sign in to see the worked answer

What the labs asked this week

Sign in to see the worked answer