Reinforcement Learning with Subspaces using Free Energy Paradigm
This work aims to accelerate reinforcement learning for large-scale problems, which is an incremental improvement for practitioners in the field.
This paper addresses the slow learning speed of reinforcement learning in large-scale problems by proposing a free-energy minimization framework for selecting and integrating state-space policies into subspaces. The framework, applicable to various tasks and state spaces, significantly improves learning speed, as demonstrated through experiments.
In large-scale problems, standard reinforcement learning algorithms suffer from slow learning speed. In this paper, we follow the framework of using subspaces to tackle this problem. We propose a free-energy minimization framework for selecting the subspaces and integrate the policy of the state-space into the subspaces. Our proposed free-energy minimization framework rests upon Thompson sampling policy and behavioral policy of subspaces and the state-space. It is therefore applicable to a variety of tasks, discrete or continuous state space, model-free and model-based tasks. Through a set of experiments, we show that this general framework highly improves the learning speed. We also provide a convergence proof.