Unraveling the Hidden Dynamical Structure in Recurrent Neural Policies
This work provides new insights into the mechanisms behind the superior generalization and robustness of recurrent policies in partially observable control and meta-RL, though it is incremental as it builds on existing dynamical system analysis without introducing a new method.
The study analyzed recurrent neural policies across various training methods, architectures, and tasks, finding that stable cyclic structures similar to limit cycles emerge during environment interaction, which stabilize internal memory and task-relevant states while suppressing environmental uncertainty.
Recurrent neural policies are widely used in partially observable control and meta-RL tasks. Their abilities to maintain internal memory and adapt quickly to unseen scenarios have offered them unparalleled performance when compared to non-recurrent counterparts. However, until today, the underlying mechanisms for their superior generalization and robustness performance remain poorly understood. In this study, by analyzing the hidden state domain of recurrent policies learned over a diverse set of training methods, model architectures, and tasks, we find that stable cyclic structures consistently emerge during interaction with the environment. Such cyclic structures share a remarkable similarity with \textit{limit cycles} in dynamical system analysis, if we consider the policy and the environment as a joint hybrid dynamical system. Moreover, we uncover that the geometry of such limit cycles also has a structured correspondence with the policies' behaviors. These findings offer new perspectives to explain many nice properties of recurrent policies: the emergence of limit cycles stabilizes both the policies' internal memory and the task-relevant environmental states, while suppressing nuisance variability arising from environmental uncertainty; the geometry of limit cycles also encodes relational structures of behaviors, facilitating easier skill adaptation when facing non-stationary environments.