Sparse neural networks perform nearly as well as dense ones. Remove 90% of activations and the model still classifies, generates, predicts. This has driven a decade of compression research: if most neurons are redundant, eliminate them and keep the capability.
The capability survives. The interpretability doesn't. This paper identifies what the authors call catastrophic interpretability collapse: under extreme sparsification, global representation quality remains stable while local feature interpretability systematically degrades. The network still works but you can no longer understand why.
The mechanism: sparsification forces the surviving neurons to encode information more densely, combining features that were previously separated across different units. A neuron that once cleanly encoded “red” now encodes a superposition of “red,” “circular,” and “large.” The representation is still there — the network can still distinguish red from blue — but no individual neuron corresponds to a human-interpretable concept.
Dead neuron rates reach 34% on simple datasets and 63% on complex ones. More complex data amplifies the collapse, because more features compete for fewer surviving neurons. The problem is algorithm-agnostic: both hard constraints (Top-k selection) and soft constraints (L1 regularization) produce similar or worse collapse. Extended training doesn't help — once interpretability is lost, it doesn't return.
The finding establishes a fundamental tradeoff that compression research has been quietly ignoring. The metrics that matter for deployment (accuracy, latency, memory) decouple from the metrics that matter for understanding (feature alignment, monosemanticity, dead neuron count). A compressed model that works perfectly and explains nothing is not a smaller version of the original. It's a different kind of object — one that performs the same function through a mechanism no one can read.