Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goals
This addresses the challenge of transferring learned information to new goal locations in tasks like robotics navigation, offering an incremental improvement over existing model-free and model-based RL methods.
The paper tackles the problem of multi-goal reinforcement learning in static environments by proposing Floyd-Warshall RL (FWRL), which adapts the Floyd-Warshall algorithm to learn a goal-conditioned action-value function, resulting in improved sample efficiency and higher reward strategies compared to baselines like Q-learning and model-based RL in tabular domains.
Consider mutli-goal tasks that involve static environments and dynamic goals. Examples of such tasks, such as goal-directed navigation and pick-and-place in robotics, abound. Two types of Reinforcement Learning (RL) algorithms are used for such tasks: model-free or model-based. Each of these approaches has limitations. Model-free RL struggles to transfer learned information when the goal location changes, but achieves high asymptotic accuracy in single goal tasks. Model-based RL can transfer learned information to new goal locations by retaining the explicitly learned state-dynamics, but is limited by the fact that small errors in modelling these dynamics accumulate over long-term planning. In this work, we improve upon the limitations of model-free RL in multi-goal domains. We do this by adapting the Floyd-Warshall algorithm for RL and call the adaptation Floyd-Warshall RL (FWRL). The proposed algorithm learns a goal-conditioned action-value function by constraining the value of the optimal path between any two states to be greater than or equal to the value of paths via intermediary states. Experimentally, we show that FWRL is more sample-efficient and learns higher reward strategies in multi-goal tasks as compared to Q-learning, model-based RL and other relevant baselines in a tabular domain.