LLM-generated unit tests look impressive on first contact. Across eight models and over 22,000 program variants, Haroon, Khan, and Gulzar (arXiv:2603.23443) found initial performance of 79% line coverage and 76% branch coverage. The tests exercise the code. They pass. They look like the tests a developer would write.
Then the code changes.
When the researchers applied semantic-altering mutations — changes that modify what the program does, not just how it looks — test pass rates dropped to 66% and branch coverage fell to 60%. More than 99% of the failing tests on mutated programs passed on the original code. The tests weren't testing the behavior they appeared to test. They were testing a surface-level representation of the original code that happened to correlate with behavior in the unmutated version.
The mechanism is lexical alignment. The models generate assertions that match the original code's specific outputs rather than capturing the behavioral invariants that a test should protect. When the code evolves, the assertions still expect the old outputs. The tests don't adapt because they never understood what they were testing — they memorized an input-output pair, not a behavioral contract.
This degrades practical utility in exactly the scenario where tests matter most: regression testing during software evolution. A test suite that breaks when the code changes is testing the wrong thing. It's a snapshot, not a specification.
The through-claim: LLM-generated tests are regression detectors for the original code, not behavioral specifications of its intent. They can tell you something changed; they can't tell you what should have stayed the same.