Self-Consistent Models and Values
This work addresses a specific bottleneck in reinforcement learning for agents that rely on environment models, offering incremental improvements over existing planning methods like Dyna.
The paper tackles the problem of improving model-based reinforcement learning by ensuring that learned models and value functions are jointly self-consistent, rather than just updating values to match the model as in classic methods. It proposes multiple self-consistency updates and finds that, with appropriate choices, these help both policy evaluation and control in tabular and function approximation settings.
Learned models of the environment provide reinforcement learning (RL) agents with flexible ways of making predictions about the environment. In particular, models enable planning, i.e. using more computation to improve value functions or policies, without requiring additional environment interactions. In this work, we investigate a way of augmenting model-based RL, by additionally encouraging a learned model and value function to be jointly \emph{self-consistent}. Our approach differs from classic planning methods such as Dyna, which only update values to be consistent with the model. We propose multiple self-consistency updates, evaluate these in both tabular and function approximation settings, and find that, with appropriate choices, self-consistency helps both policy evaluation and control.