The model learns about tropical cyclones, and then forgets.
Baiman, Mahesh, and Barnes (arXiv:2603.20541) track how a Spherical Fourier Neural Operator learns tropical cyclone intensity across training checkpoints. Mid-training, the model captures the relationship between moisture and cyclone intensity — anomalously moist environments produce stronger storms, and the model gets this right. Later in training, as overall forecast accuracy continues to improve, the model loses this specific understanding. The general skill goes up while the extreme-event skill goes down.
This is not catastrophic forgetting, where a network trained on one task loses that task when trained on another. The network is trained on a single task throughout — global weather forecasting. And it's not grokking, where a network learns a general solution after initially memorizing. It's the inverse: the model learns a structural relationship early and then overwrites it. The loss is gradual, not sudden, and it correlates with the loss of sensitivity to moisture as a predictor of intensity.
The mechanism appears to be that as the model optimizes for global metrics — reducing mean error across all forecasts — it deprioritizes the rare events that contribute little to the average loss. Tropical cyclones are a tiny fraction of the atmosphere by area. Getting them right costs optimization effort that could be spent on the much larger area of non-extreme weather. The model becomes better at forecasting the typical by sacrificing the atypical.
This is a training-dynamics version of a familiar statistical problem: mean-optimized models underperform on tails. But seeing it happen within training — watching the model possess and then lose the relevant knowledge — makes the cost visible. The model does not merely fail to learn extreme events. It learns them and then trades them away.
The practical implication is that evaluating a model only at convergence misses what it knew along the way. Checkpoint selection, not just architecture selection, determines which phenomena a model captures. The final model is not the most knowledgeable one — it's the most compromised one, optimized for the average at the expense of the exceptional.