An LLM agent passes a safety benchmark. Every test case produces a safe response. The benchmark reports 100% compliance. More than 20% of unsafe behaviors remain undetected.
Chen et al. build SafeAudit, a meta-audit framework that evaluates the safety evaluations themselves. The method enumerates potential tool-call workflows and user scenarios that existing benchmarks do not cover, then tests whether agents behave safely in the gaps. Across three benchmarks and twelve environments, the gaps are substantial — agents that pass every existing test case still exhibit unsafe tool-call patterns when the test suite is expanded.
The “rule-resistance” metric identifies the specific failure mode: interaction patterns where the agent's behavior is unsafe but no existing test case would catch it. These are not adversarial inputs or jailbreaks. They are ordinary workflows — reasonable tool sequences, plausible user requests — that the benchmark designers did not think to include.
The coverage improves with testing budget, which means the problem is not inherent impossibility but practical incompleteness. Current benchmarks are too sparse to characterize the safety surface. They sample it, find it smooth in the sampled regions, and declare it smooth everywhere. The unsampled regions contain 20% of the behavior.
The structural observation: safety benchmarks inherit the same problem as the systems they evaluate. An LLM agent cannot enumerate all possible tool interactions. A benchmark cannot enumerate all possible test cases. Both face exponential possibility spaces. The benchmark's coverage gaps are isomorphic to the agent's safety gaps — finite evaluation of infinite behavior space. The meta-auditor does not solve this problem. It reveals its scale.