Automated program repair generates candidate patches for bugs. Most candidates overfit — they pass the test suite but remain incorrect. Six state-of-the-art tools detect overfitting patches: static analysis, dynamic testing, machine learning classifiers. Each claims substantial accuracy on published benchmarks.
Namavar, Ebrahimi, and Haghighi (arXiv:2603.11262, 2026) run the same tools under realistic conditions. The result: simple random selection outperforms all six tools in 71% to 96% of cases, depending on the tool.
The tools are not broken. The benchmarks are. Published evaluations use curated patch distributions — carefully balanced sets where the ratio of correct to overfitting patches is far higher than what real repair tools produce. When the base rate shifts to realistic levels — where correct patches are rare and overfitting patches dominate — every tool's performance degrades below random. The tools learned to discriminate patches from a distribution they will never encounter in practice.
This is not the familiar overfitting story where a model memorizes training data. The tools may generalize well within the curated distribution. The failure is that the distribution itself was wrong. The curation that made evaluation tractable also made it uninformative. The researchers didn't fail to generalize from the data. They generalized perfectly from data that didn't represent the problem.
The mechanism is base-rate mismatch. When correct patches are common (curated setting), even a modest ability to distinguish correct from overfitting provides value. When correct patches are rare (realistic setting), the same tool's false positive rate overwhelms its true positive rate. Random selection, which has no pretension to discrimination, suffers the same base rate but imposes no systematic bias. It wins not by being smarter but by being no dumber than the problem demands.
Six tools, multiple methodologies, years of research — and the collective output is worse than not trying. The field did not lack effort or sophistication. It lacked a baseline that matched the deployment distribution. The curated benchmark created the appearance of progress by solving a version of the problem that does not exist in practice.