Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
This work addresses the challenge of understanding and enhancing neural mechanisms in RL for continuous control, though it appears incremental in advancing RNN architectures.
The paper tackled the problem of improving recurrent neural networks for reinforcement learning by proposing a multiple-timescale stochastic RNN, which autonomously learned to abstract sub-goals and develop an action hierarchy in a continuous control task, resulting in enhanced faster re-learning for new tasks.
Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical results show that the network can autonomously learn to abstract sub-goals and can self-develop an action hierarchy using internal dynamics in a challenging continuous control task. Furthermore, we show that the self-developed compositionality of the network enhances faster re-learning when adapting to a new task that is a re-composition of previously learned sub-goals, than when starting from scratch. We also found that improved performance can be achieved when neural activities are subject to stochastic rather than deterministic dynamics.