Deep transformers have a signal degradation problem. A sharp feature formed at layer 5 gets diluted by residual updates through layers 6, 7, 8, ... each adding their own information and noise. By layer 50, the original feature is faint — not lost (the residual connection preserves it) but buried under accumulated updates. The deeper the model, the harder it is for late layers to access early features.
Mixture-of-depths attention (MoDA) adds skip connections in the attention mechanism. Each attention head can attend not just to the current layer's key-value pairs but also to those from earlier layers. The head learns which depth to query: sometimes the current-layer representation is best, sometimes the answer is in an earlier, less processed version.
The implementation matters as much as the idea. Naively attending to multiple layers multiplies the key-value cache and destroys the efficiency that makes transformers practical. MoDA achieves near-FlashAttention-2 efficiency through a hardware-aware implementation that keeps the multi-depth lookup within the memory bandwidth constraints of modern GPUs.
The results at 1.5B parameters: 0.2 perplexity improvement across 10 benchmarks and 2.11% improvement on downstream tasks, for a 3.7% increase in FLOPs. The gains come from combining MoDA with post-norm (layer normalization after the residual, not before). Post-norm amplifies the benefit because it doesn't normalize away the skip-connection signal — the very signal MoDA is designed to recover.
Depth as a queryable resource. The model doesn't just stack layers — it reaches back through them, choosing the right processing depth for each feature.