At 100,000 GPUs, failures during LLM pre-training are not exceptions — they are the dominant cost. Restart overhead eclipses useful computation, and traditional fault tolerance through replication scales linearly: doubling redundancy doubles the computational waste. SPARe breaks this relationship by stacking redundant data shards across parallelism groups and adaptively reordering execution when nodes fail. The approach masks failures during gradient synchronization rather than halting and restarting, maintaining near-constant overhead of 2-3x regardless of redundancy level — where traditional replication would inflate costs linearly.
The key architectural insight is the separation of redundancy from overhead. In conventional replication, every redundant copy computes in lockstep, wasting resources proportional to the redundancy factor. SPARe achieves the same availability guarantees by reorganizing computation on the fly, so redundant shards only activate when needed. The closed-form expressions for endurable failure count and overhead enable joint optimization with checkpointing, reducing time-to-train by 40-50% at 600,000 GPUs compared to standard approaches.
The broader principle: at sufficient scale, resilience cannot be achieved by doing more of the same. Linear redundancy strategies break down when the failure rate is high enough that redundant resources spend most of their time duplicating work that will complete successfully anyway. Adaptive reorganization — changing the structure of computation in response to failure — achieves what brute-force replication cannot.
(arXiv:2603.00357)