Recurrent neural networks exhibit chaotic dynamics when the weight matrices are large enough — the Lyapunov exponents become positive, trajectories diverge exponentially, and long-term predictions become impossible. This is the edge-of-chaos phenomenon: networks are most expressive near the chaotic transition but least stable there.
The paper identifies a class of recurrent architectures that are provably chaos-free for any weight values. The key: structural constraints on the weight matrix that ensure all Lyapunov exponents are non-positive, regardless of the activation function, the input, or the network's dimension. The constraints are not on the magnitude of the weights (which would limit expressivity) but on their algebraic structure (which preserves expressivity while guaranteeing stability).
The chaos-free networks are stable in the Lyapunov sense: nearby trajectories never diverge, perturbations never amplify, and the network's long-term behavior is well-defined. This stability is unconditional — it holds for all inputs and all time horizons, not just in expectation or on average.
The expressivity trade-off: chaos-free networks cannot approximate arbitrary dynamical systems (some systems are chaotic, and a chaos-free network can't be chaotic). But they can approximate any stable dynamical system to arbitrary precision, which is the relevant class for most control and prediction tasks. The networks are expressive where they need to be (stable systems) and unable to be where they shouldn't be (chaotic systems).
Stability by construction, not by regularization. The architecture prevents chaos; the training optimizes within the stable regime. The constraint isn't a cost — it's a feature that makes the network's behavior predictable at any time horizon.