Reusability and Transferability of Macro Actions for Reinforcement Learning
This work addresses the problem of improving learning efficiency in RL by leveraging macro actions, but it appears incremental as it focuses on analyzing existing properties rather than introducing new methods.
The paper investigates the beneficial properties of macro actions in reinforcement learning, specifically reusability and transferability, and provides experimental analyses to reveal these properties.
Conventional reinforcement learning (RL) typically determines an appropriate primitive action at each timestep. However, by using a proper macro action, defined as a sequence of primitive actions, an agent is able to bypass intermediate states to a farther state and facilitate its learning procedure. The problem we would like to investigate is what associated beneficial properties that macro actions may possess. In this paper, we unveil the properties of reusability and transferability of macro actions. The first property, reusability, means that a macro action generated along with one RL method can be reused by another RL method for training, while the second one, transferability, means that a macro action can be utilized for training agents in similar environments with different reward settings. In our experiments, we first generate macro actions along with RL methods. We then provide a set of analyses to reveal the properties of reusability and transferability of the generated macro actions.