Learning Dynamics of RNNs in Closed-Loop Environments
This work addresses the gap in biologically plausible learning models for neuroscience-inspired tasks, though it is incremental as it builds on existing RNN frameworks.
The paper tackled the problem of training recurrent neural networks (RNNs) in closed-loop environments, unlike typical open-loop settings, by developing a mathematical theory for linear RNNs and showing that closed-loop training leads to different learning trajectories governed by competing objectives like policy improvement and stability.
Recurrent neural networks (RNNs) trained on neuroscience-inspired tasks offer powerful models of brain computation. However, typical training paradigms rely on open-loop, supervised settings, whereas real-world learning unfolds in closed-loop environments. Here, we develop a mathematical theory describing the learning dynamics of linear RNNs trained in closed-loop contexts. We first demonstrate that two otherwise identical RNNs, trained in either closed- or open-loop modes, follow markedly different learning trajectories. To probe this divergence, we analytically characterize the closed-loop case, revealing distinct stages aligned with the evolution of the training loss. Specifically, we show that the learning dynamics of closed-loop RNNs, in contrast to open-loop ones, are governed by an interplay between two competing objectives: short-term policy improvement and long-term stability of the agent-environment interaction. Finally, we apply our framework to a realistic motor control task, highlighting its broader applicability. Taken together, our results underscore the importance of modeling closed-loop dynamics in a biologically plausible setting.