Causally Correct Partial Models for Reinforcement Learning
This addresses a key issue in RL for agents using partial models, offering a solution to improve planning accuracy without high computational costs.
The paper tackles the problem of partial models in reinforcement learning being causally incorrect due to unmodeled observations, which can lead to flawed planning. It introduces a family of causally correct partial models that avoid full observation modeling, maintaining computational efficiency.
In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent's next actions. However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images). For this reason, previous works have considered partial models, which model only part of the observation. In this paper, we show that partial models can be causally incorrect: they are confounded by the observations they don't model, and can therefore lead to incorrect planning. To address this, we introduce a general family of partial models that are provably causally correct, yet remain fast because they do not need to fully model future observations.