Neural tangent kernel (NTK) theory promises that sufficiently wide networks behave like kernel methods — the kernel is fixed at initialization and the dynamics are linear. This makes analysis tractable. For ReLU networks, moderate overparameterization suffices for convergence. For attention, it doesn't.
Wang and Zhu (arXiv:2603.13085) show that linearized attention cubes the condition number of the Gram matrix. If the data has condition number κ, convergence to the NTK limit requires width m = Ω(κ⁶). On natural image datasets where κ is already large, this means impractically wide networks — thousands of times wider than what suffices for ReLU.
The non-convergence isn't a failure. It's a feature viewed from the wrong theory. When attention doesn't converge to its NTK limit, it means the kernel is still changing during training — the network is dynamically adjusting which training examples influence the prediction at each input. The authors call this influence malleability: attention networks show 6-9x higher malleability than ReLU networks of comparable width.
The dual consequence: high malleability lets attention align its effective kernel with the task structure, reducing approximation error compared to a fixed kernel. The same malleability makes the model vulnerable to adversarial manipulation of the training data — poisoning a few examples has outsized influence precisely because the model is designed to dynamically adjust its reliance on them.
Attention's effectiveness and its vulnerability share a common origin: departure from the kernel regime. The same mechanism that makes attention powerful makes it fragile. The NTK framework doesn't fail to describe attention — it successfully identifies exactly why attention is different.