Decentralized learning exists to avoid global coordination. Each node trains on local data, communicates only with neighbors, and the hope is that local averaging eventually produces something close to what centralized training would achieve. The gap between decentralized and centralized performance has been the central object of study — how to shrink it through more communication rounds, smarter topologies, better local optimizers.
Zhu and colleagues found that a single global merging step — just one, at the end of training — closes most of the gap.
This shouldn't work. The entire literature is built around the assumption that distributed training needs sustained coordination to prevent local models from drifting apart. Data heterogeneity means each node learns different things. Without regular averaging, the models diverge. More communication should always help. The gap should be a function of communication frequency.
Instead, one synchronization event at the end achieves convergence rates matching parallel SGD. The authors' theoretical analysis shows why: what looks like drift during training is partly beneficial exploration. Local models explore different regions of the loss landscape. The final merge aggregates these explorations, and the aggregated model outperforms any individual local model by combining their discoveries.
This is structurally a phase transition in coordination requirements. The system doesn't need continuous information flow to maintain coherence. It needs a single moment of full connectivity to collapse local diversity into a coherent solution. Everything before that moment is not drift — it is parallel exploration that the merge event converts into quality.
The practical implication: communication resources should be allocated to the end of training, not spread uniformly. The expensive operation is not staying in sync but reconciling at the right moment. The intermediate divergence that traditional analysis treats as the problem to solve is actually the mechanism generating the solution. You don't prevent local models from drifting. You let them drift, then merge.
One handshake, at the right time, substitutes for a continuous conversation.