Neural networks initialized at the “edge of chaos” — the boundary between ordered and chaotic signal propagation — train best. This is an empirical observation, well-established across architectures. The usual explanation: the edge of chaos balances signal preservation against signal diversity, providing enough structure for learning without rigidity.
The observation is correct. The explanation is incomplete. The edge of chaos in neural networks is not merely analogous to a phase transition. It is a phase transition, and it belongs to a specific universality class from non-equilibrium statistical mechanics.
Multilayer perceptrons at the edge of chaos exhibit the scaling exponents of the mean-field absorbing phase transition universality class. Convolutional neural networks exhibit the scaling exponents of the directed percolation universality class. These are not metaphors. The critical exponents — the numbers that describe how correlation length, order parameter, and susceptibility scale near the transition — match the known values from statistical mechanics within measurement precision.
The distinction matters because universality classes are not just classifications. They are predictions. If a system belongs to directed percolation, then its critical behavior is completely determined by the universality class, regardless of microscopic details. The macro-scale behavior near the transition depends on dimension, symmetry, and conservation laws — not on the specific architecture, activation function, or loss landscape of the network.
This means the edge-of-chaos initialization heuristic is not a trick. It is placing the network at a genuine critical point whose properties are known from decades of statistical mechanics research. The generalization properties near this point, the sensitivity to hyperparameters, the training dynamics — these are governed by critical exponents that were computed for entirely different physical systems (epidemic spreading, magnetic ordering) long before deep learning existed.
Architecture choice is, in part, a choice of universality class. An MLP and a CNN do not merely have different inductive biases. They live in different universality classes, with different critical exponents, different correlation structures, different finite-size scaling. The architecture determines which phase transition the network sits at, and the phase transition determines the learning dynamics.
The physics was already solved. We just didn't know we were using it.