Reinforcement learning algorithms are tested on environments where the optimal policy is unknown. Performance is measured by comparison — agent A scores higher than agent B — but never against ground truth. When one agent outperforms another on CartPole, the question of how close either is to optimal remains unanswered. Benchmarking without a known optimum measures relative performance, not absolute capability.
Ibrahim et al. invert the problem. Instead of solving for the optimal policy in a given environment, they prescribe the optimal policy first, then derive the environment. Converse optimality theory provides the conditions under which a given value function and policy are optimal for some cost function. Given these conditions, you can construct environments with known ground-truth optimal policies, then test RL algorithms against the actual optimum rather than against each other.
The framework applies to discrete-time, control-affine, nonlinear systems with noise — not toy problems. The generated environments are diverse because the mapping from (policy, value function) to environment is many-to-one: different environments can share the same optimal policy, and different policies can be optimal for different costs on the same dynamics.
The structural point: the hardest part of benchmarking is not building harder environments. It is knowing what optimal looks like. Converse optimality supplies this by construction rather than computation. The inversion — from “find the policy for this environment” to “find the environment for this policy” — transforms an intractable verification problem into a generative one. You don't solve for the answer; you build the question around the answer you already have.