Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning
This addresses a key problem in training goal-conditioned RL agents for reliable goal-reaching and generalization, with incremental improvements in representation methods.
The paper tackles the challenge of specifying and grounding goals in noisy, high-dimensional sensory inputs for goal-conditioned reinforcement learning by learning factorial representations with a discretization bottleneck, called DGRL, and shows improved performance in tasks like maze environments and robotic navigation/manipulation, while providing a theoretical lower bound on expected return for out-of-distribution goals.
Goal-conditioned reinforcement learning (RL) is a promising direction for training agents that are capable of solving multiple tasks and reach a diverse set of objectives. How to \textit{specify} and \textit{ground} these goals in such a way that we can both reliably reach goals during training as well as generalize to new goals during evaluation remains an open area of research. Defining goals in the space of noisy and high-dimensional sensory inputs poses a challenge for training goal-conditioned agents, or even for generalization to novel goals. We propose to address this by learning factorial representations of goals and processing the resulting representation via a discretization bottleneck, for coarser goal specification, through an approach we call DGRL. We show that applying a discretizing bottleneck can improve performance in goal-conditioned RL setups, by experimentally evaluating this method on tasks ranging from maze environments to complex robotic navigation and manipulation. Additionally, we prove a theorem lower-bounding the expected return on out-of-distribution goals, while still allowing for specifying goals with expressive combinatorial structure.