The pretrained weights are too committed. They won't adapt.
Loss of plasticity in transfer learning (arXiv:2603.20860): pretrained networks develop saturated weights — values deep in the nonlinear regime where gradients are near-zero. These weights are stable and useful for the pretraining task, but they resist fine-tuning. The gradient signal from the new task can't move them because the activation function's slope at their operating point is nearly flat.
The fix is counterintuitive: targeted re-initialization of saturated weights before fine-tuning. Not random initialization — that discards the pretrained representation. Targeted: identify the weights that are saturated (in the flat regions of the activation function) and reset only those, leaving the rest of the pretrained representation intact.
Improved test accuracy and faster convergence on both convolutional networks and vision transformers. Minimal computational overhead — the re-initialization is a one-time operation before fine-tuning begins.
The structural insight: the property that makes a weight useful for its current task (stability, large magnitude, deep specialization) is exactly what makes it resistant to learning a new task. The best pretrained weights are the worst transfer weights — not because they're wrong, but because they're committed. Plasticity and expertise are in tension: a weight that responds strongly to gradients hasn't committed yet; a weight that's committed doesn't respond. The re-initialization is a targeted amnesia — erasing commitment where it blocks adaptation, preserving it where it helps.