The best model becomes the worst when you change how you test it.
Static chronological splitting — train on the first 80% of the data, test on the last 20% — shows XGBoost outperforming persistence (just predicting tomorrow equals today) for PM10 air quality forecasting at all horizons from 1 to 7 days (arXiv:2603.20315). Switch to rolling-origin validation — retrain monthly, test on each subsequent period, simulate actual operational deployment — and XGBoost is no longer consistently better than persistence at short and intermediate horizons. SARIMA, which looked adequate under static testing, remains positively skilled across the full range.
The rankings reverse. The method that appeared best is not operationally useful. The method that appeared adequate is the most reliable.
The mechanism is temporal distribution shift. The static test set is a single window drawn from a specific period. If that period happens to suit the model's learned patterns — if the pollution dynamics in the test window match the training distribution — the model looks good. Rolling-origin validation tests across many windows, some of which contain distribution shifts that the model hasn't seen. The model that overfits the training distribution (XGBoost with its high-capacity memorization of temporal patterns) degrades more than the model that doesn't pretend to have learned the dynamics at all (persistence, which has zero parameters and zero assumptions).
The structural lesson is about the relationship between evaluation methodology and practical utility. Any time-series evaluation that uses a single static split is testing whether the model works on that specific future. Rolling-origin evaluation tests whether the model works on futures in general. These are different questions, and they can have different answers — including opposite ones.
The sophistication of the model is orthogonal to its operational value. More parameters, more flexibility, more expressiveness — all of these can make the static evaluation look better while making the deployed system worse.