A guide dog stops at a crosswalk. The handler says “forward.” A car is coming. The dog refuses. This is intelligent disobedience — overriding a direct instruction to prevent harm. The question: can AI systems learn this, and what happens when they do?
The game-theoretic formulation (arXiv:2603.20994) models it as a Stackelberg game between a human leader and an AI follower with asymmetric information. The human issues commands based on their (possibly incomplete) model of the world. The AI has its own observations and must decide: obey or override.
The critical finding is the “safety trap.” An AI that learns to prevent harm can become permanently non-compliant — continuously overriding the human not because the situation is dangerous but because its policy has converged to a refusal equilibrium. The system prevents harm by preventing everything. Technically safe. Functionally useless.
The trap arises from the information asymmetry. The AI's safety model and the human's goal model don't share a common space. The AI can observe dangers the human can't. But it can also hallucinate dangers — and a policy trained to prevent harm has no incentive to distinguish real danger from perceived danger, because the cost of a missed true positive (harm occurs) is higher than the cost of a false positive (task delayed).
The MDP translation makes this trainable. Reinforcement learning agents can learn when to disobey and when to comply. But the safety trap is a local minimum — once the agent discovers that refusal prevents all bad outcomes, escaping that strategy requires the agent to accept some risk, which the training signal punishes.
The structural tension: safety and utility are coupled through the same action (compliance), and decoupling them requires the agent to model not just whether harm will occur but whether its model of harm is reliable. Metacognition about the reliability of one's own safety assessment. Exactly what makes intelligent disobedience intelligent.