Hybrid language models combine attention mechanisms with state space models (SSMs) or linear attention. The promise: each mechanism handles what the other can't. Attention captures long-range dependencies; SSMs process sequences efficiently. But do both components actually do meaningful work, or does one dominate while the other coasts?
Borobia, Seguí-Mas, and Tormo-Carbó (arXiv:2603.22473) ablate components systematically in two hybrid models (Qwen3.5-0.8B and Falcon-H1-0.5B) alongside a pure Transformer baseline. The findings: both component types are essential, neither is bypassed, but they are not equal. The SSM/linear attention component functions as the primary backbone — removing it causes larger performance drops than removing the attention heads. Attention is important but secondary.
The gradient of importance decreases from early to later layers, suggesting the architectural division of labor shifts with depth. Early layers need both mechanisms; later layers are more robust to either being removed.
The most striking result is resilience. Hybrid architectures show 20-119x greater robustness to random layer removal compared to pure Transformers. The functional redundancy between attention and SSM components means that when one layer is damaged or removed, the other mechanism in surrounding layers can partially compensate. The pure Transformer has no such backup.
The structural lesson is that architectural diversity buys robustness through redundancy, not just through complementary capabilities. The hybrid model is more fault-tolerant not because each component is better at its specific task, but because having two different mechanisms doing partially overlapping work means neither is a single point of failure. Specialization and redundancy coexist because the components are different enough to specialize but similar enough to compensate.