The attack works in the paper. It fails in the deployment.
A systematic evaluation of federated learning poisoning attacks (arXiv:2603.20615) finds that idealized research conditions consistently overstate security risk. Attacks that appear effective in standard benchmarks lose their potency — or cause unacceptable collateral damage to benign tasks — when tested under real-world constraints.
The mismatches are specific. Research assumes unrealistic attacker participation rates (higher fractions of compromised clients than any practical system would allow). Studies ignore hybrid heterogeneity (real FL systems have clients with different data distributions, hardware, and connectivity). Metrics emphasize peak attack success at the final training round while ignoring temporal stability (an attack that works once but fails in surrounding rounds isn't operationally useful).
The most damaging finding: several attacks appearing consistently effective in idealized settings incur clear benign-task degradation once practical constraints are enforced. The attack “works” only if you ignore what it breaks. Joint evaluation — effectiveness, stability, and collateral damage — reverses the conclusions of papers that report each metric in isolation.
The structural problem is that security research optimizes for publishable threat demonstrations, not for accurate risk assessment. A dramatic attack in an idealized setting is more publishable than a nuanced analysis showing the attack is impractical. The field selects for alarming results and against calibrated ones.
This doesn't mean FL is secure. It means we don't know how insecure it is, because the measurement instrument (idealized evaluation) is systematically biased toward overestimation. The defense literature inherits the same bias — if the threats are overstated, the defenses are solving the wrong problem. Accurate risk requires matching the evaluation to the deployment, and the gap between lab and field is where the real security posture hides.