How Hard is it to Confuse a World Model?
This work addresses a theoretical gap in RL for constructing confusing instances, which is incremental as it extends known concepts to neural network models.
The paper tackles the problem of constructing most confusing instances for neural network world models in reinforcement learning, formalizing it as a constrained optimization and proposing an adversarial training procedure. The results show that the degree of achievable confusion correlates with uncertainty in the approximate model, potentially informing exploration strategies for deep model-based RL.
In reinforcement learning (RL) theory, the concept of most confusing instances is central to establishing regret lower bounds, that is, the minimal exploration needed to solve a problem. Given a reference model and its optimal policy, a most confusing instance is the statistically closest alternative model that makes a suboptimal policy optimal. While this concept is well-studied in multi-armed bandits and ergodic tabular Markov decision processes, constructing such instances remains an open question in the general case. In this paper, we formalize this problem for neural network world models as a constrained optimization: finding a modified model that is statistically close to the reference one, while producing divergent performance between optimal and suboptimal policies. We propose an adversarial training procedure to solve this problem and conduct an empirical study across world models of varying quality. Our results suggest that the degree of achievable confusion correlates with uncertainty in the approximate model, which may inform theoretically-grounded exploration strategies for deep model-based RL.