Design fault-tolerant training for a frontier model on 16k accelerators

A 400B-parameter model trains on about 16,000 accelerators for two months. At that scale hardware fails every few hours and each failure stops the synchronous job. Design the system that keeps the run going: how the model is split across chips, how training state is checkpointed and restored exactly, how failures, stragglers, loss spikes and silent data corruption are detected and handled, and how much compute is lost. Work out the failure rate, checkpoint size and the best checkpoint interval.