Convergence and stability of Q-learning in Hierarchical Reinforcement Learning
This provides a principled theoretical foundation for Feudal RL, addressing a gap in the field, though it is incremental as it builds on existing Q-learning and hierarchical methods.
The paper tackles the lack of theoretical guarantees for Hierarchical Reinforcement Learning by proposing a Feudal Q-learning scheme and analyzing its convergence and stability conditions, showing that updates converge to an equilibrium interpretable as a game-theoretic solution.
Hierarchical Reinforcement Learning promises, among other benefits, to efficiently capture and utilize the temporal structure of a decision-making problem and to enhance continual learning capabilities, but theoretical guarantees lag behind practice. In this paper, we propose a Feudal Q-learning scheme and investigate under which conditions its coupled updates converge and are stable. By leveraging the theory of Stochastic Approximation and the ODE method, we present a theorem stating the convergence and stability properties of Feudal Q-learning. This provides a principled convergence and stability analysis tailored to Feudal RL. Moreover, we show that the updates converge to a point that can be interpreted as an equilibrium of a suitably defined game, opening the door to game-theoretic approaches to Hierarchical RL. Lastly, experiments based on the Feudal Q-learning algorithm support the outcomes anticipated by theory.