Operator World Models for Reinforcement Learning
This work addresses a theoretical bottleneck in reinforcement learning for researchers and practitioners, offering a novel method with proven optimality, though it appears incremental in building on existing PMD frameworks.
The paper tackles the challenge of applying Policy Mirror Descent to reinforcement learning by introducing a world model based on conditional mean embeddings, deriving a closed-form action-value function, and proposing the POWR algorithm, which achieves proven convergence rates to the global optimum.
Policy Mirror Descent (PMD) is a powerful and theoretically sound methodology for sequential decision-making. However, it is not directly applicable to Reinforcement Learning (RL) due to the inaccessibility of explicit action-value functions. We address this challenge by introducing a novel approach based on learning a world model of the environment using conditional mean embeddings. Leveraging tools from operator theory we derive a closed-form expression of the action-value function in terms of the world model via simple matrix operations. Combining these estimators with PMD leads to POWR, a new RL algorithm for which we prove convergence rates to the global optimum. Preliminary experiments in finite and infinite state settings support the effectiveness of our method