A language model with hundreds of billions of parameters and hundreds of layers should have complex internal dynamics. Each layer transforms the representation, and the composition of hundreds of nonlinear transformations should produce behavior that resists simple description. Complexity breeds complexity — that's the intuition.
Nettasinghe and Joseph (arXiv:2603.12541, March 2026) measure the layer-by-layer dynamics of GPT-2-large across multiple tasks — toxicity detection, irony, sentiment analysis — and find that a 32-dimensional linear dynamical system reproduces the model's depth behavior accurately.
The model has 774 million parameters. Its depth dynamics fit in 32 dimensions. And the accuracy of the linear fit improves as models get larger. More parameters, more layers, more potential complexity — and simpler dynamics.
The mechanism is regularization through scale. Larger models have more capacity to distribute computation across layers. When capacity is abundant, each layer makes small, incremental changes to the representation. Small changes compose linearly to a good approximation. The depth trajectory becomes a smooth curve in a low-dimensional subspace, not because the model is simple, but because it has enough capacity to be smooth.
The practical consequence is control-theoretic intervention. If the depth dynamics are linear, established tools from control theory — state feedback, observer design, optimal control — apply directly. Steering a model's internal representation toward or away from a target (reducing toxicity, adjusting sentiment) becomes a standard control problem with known solutions. The interventions require less energy (fewer parameter modifications) than heuristic approaches because they exploit the linear structure.
The structural lesson: complexity in the parts doesn't imply complexity in the whole. The individual layers are nonlinear. The composition of hundreds of nonlinear layers produces linear dynamics in the aggregate. The emergence of simplicity from complexity is not an approximation — it's the actual behavior of the system at sufficient scale. Bigger is simpler, when “simpler” means “more predictable by low-dimensional surrogates.”