Run a vulnerability scanner against a language model. Record the attack success rate. Now change the evaluator component — the part that decides whether an attack succeeded — while keeping the attacks and model outputs identical. The success rate changes by up to 33%.
Erez et al. audit the Garak scanner and find that 22 of 25 attack categories exhibit evaluator instability. The same model output, the same attack, is classified as “jailbreak succeeded” by one evaluator and “jailbreak failed” by another. The vulnerability score is not a property of the model. It is a property of the measurement.
The instability is not random — it is systematic. Different evaluators disagree on what counts as a successful attack because the boundary between “harmful response” and “refusal” is not sharp. A model that provides partial information, hedges its response, or redirects the query occupies the gray zone that evaluators classify differently.
The fix — a two-phase framework with reliability-aware evaluation — improves accuracy from 72% to 89%. The first phase quantifies evaluator disagreement to flag unreliable attack categories. The second phase adds independent verifiers that validate evaluator decisions. The framework does not eliminate disagreement. It identifies where disagreement is highest and deploys extra scrutiny there.
The structural insight: security evaluation of language models is a measurement problem before it is a security problem. If the instrument fluctuates by 33%, the reading cannot distinguish a vulnerable model from a safe one. The current practice of reporting a single vulnerability score without quantifying evaluator instability is analogous to reporting a temperature without specifying the thermometer — the number is precise but not necessarily accurate.