Transformer attention scales as O(n²d) — quadratic in sequence length, linear in feature dimension. Most efficiency research attacks the n² term: sparse attention patterns, linear approximations, sliding windows. The d dimension is left alone.
Fan et al. (arXiv:2603.22300) attack d instead. Sparse Feature Attention represents queries and keys as k-sparse codes — only k of d features are nonzero. The attention computation drops from O(n²d) to O(n²k²/d), because sparse inner products only need to consider overlapping nonzero features. A custom kernel (FlashSFA) avoids materializing dense matrices.
GPT-2 and Qwen3 pretraining show parity with dense attention at 2.5x speedup and nearly 50% FLOP reduction. The KV-cache, which dominates memory during inference, shrinks proportionally. Unlike compression methods that reduce n (and lose long-range information), sparse features maintain full sequence access while reducing the cost of each attention operation.
The through-claim: the attention bottleneck has two axes, and the research community has been optimizing the wrong one. Sequence length gets most of the attention (pun intended), but feature sparsity is equally effective and doesn't sacrifice the global connectivity that makes transformers powerful. The most effective optimization was hiding in the dimension nobody was looking at.