Twenty-nine RNA secondary structure prediction methods were evaluated. The foundation models won. Then the benchmark was changed — not the models, not the training data, just the test set — and the foundation models lost.
Chen, Deng, Deng, Liao, Su, Ye, and Liu (arXiv:2603.22330) built CHANRG: 170,083 structurally non-redundant RNAs from over 10 million Rfam sequences, split with genome-aware design so that test families are genuinely absent from training. On the held-out data (drawn from the same distribution as training), foundation models achieved the highest accuracy. Out of distribution — tested on RNA families they'd never seen — they lost most of that advantage. Structured decoders and direct neural predictors, which encode base-pairing constraints explicitly, remained markedly more robust.
The mechanism is the usual one: foundation models memorize sequence-level regularities. When those regularities hold, memorization outperforms physics. When they don't — when the RNA family is genuinely novel — the memorized patterns fail, and the models that encode the physical rules of base-pairing (which hold regardless of family) maintain their accuracy.
What's structural here is that the leaderboard didn't just shift. It flipped. The method that ranked first in-distribution didn't merely drop a few places out-of-distribution; it fell below methods it had dominated. The ranking isn't a property of the models. It's a property of the test.
This has a formal name in decision theory: the ranking function is not monotone across evaluation domains. A method that's better on task A can be worse on task B without either method changing. The benchmarks that fail to enforce domain separation hide this non-monotonicity — they report a single ranking that averages over the regime where memorization works and the regime where it doesn't, producing a number that describes neither regime accurately.
The practical implication is that every leaderboard implicitly encodes a belief about what “new” means. If “new” means a held-out sample from the same distribution, memorization is the optimal strategy. If “new” means a genuinely novel structure, encoding the physics is. The benchmark doesn't measure which model is better. It measures which definition of “new” you're using.