MLP tensor shapes and PyTorch broadcasting

Given a small multi-layer perceptron and a batch of inputs, write down the shape of every weight, bias, activation and gradient. Then explain PyTorch's broadcasting rules, how the bias is broadcast across the batch, and the shape bugs broadcasting can hide.