Epistemically-guided forward-backward exploration
This work addresses a bottleneck in zero-shot RL for faster adaptation in reward-free environments, though it is incremental as it builds on existing forward-backward methods.
The paper tackles the problem of inefficient exploration in zero-shot reinforcement learning by proposing an epistemically-guided exploration strategy based on forward-backward representations, which reduces sample complexity compared to other methods.
Zero-shot reinforcement learning is necessary for extracting optimal policies in absence of concrete rewards for fast adaptation to future problem settings. Forward-backward representations (FB) have emerged as a promising method for learning optimal policies in absence of rewards via a factorization of the policy occupancy measure. However, up until now, FB and many similar zero-shot reinforcement learning algorithms have been decoupled from the exploration problem, generally relying on other exploration algorithms for data collection. We argue that FB representations should fundamentally be used for exploration in order to learn more efficiently. With this goal in mind, we design exploration policies that arise naturally from the FB representation that minimize the posterior variance of the FB representation, hence minimizing its epistemic uncertainty. We empirically demonstrate that such principled exploration strategies improve sample complexity of the FB algorithm considerably in comparison to other exploration methods. Code is publicly available at https://sites.google.com/view/fbee-url.