In the soft committee machine — a two-layer neural network with a teacher-student setup — hidden units start interchangeable (symmetric) and must specialize to learn. In sigmoidal networks, this specialization transition is first-order: the hidden units are stuck in symmetry until a critical sample size, then snap into specialized roles.
The authors (arXiv:2603.20010) show that with ReLU activation, the transition is fundamentally different — continuous, not first-order. The hidden units gradually differentiate as sample size increases, without the stuck phase that plagues sigmoidal networks.
The through-claim: the activation function determines the nature of the learning transition, not just the convergence rate. ReLU's piecewise linearity creates a smoother energy landscape where partial specialization is stable. Sigmoidal activations create a symmetric minimum that traps the system until enough data breaks the symmetry catastrophically. The practical implication: ReLU networks don't suffer from the symmetry-trapping plateau that makes sigmoidal committee machines hard to train, and this difference has a precise statistical-mechanical explanation.