
Design fault-tolerant training for a frontier model on 16k acceleratorsSystem designA 400B-parameter model trains on about 16,000 accelerators for two months. At that scale hardware fails every few hours and each failure stops the synchronous job. Design the…