Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning
This work addresses the challenge of building more scalable and compressible artificial networks, with potential applications in evolutionary learning of complex tasks, though it is incremental in exploring modularity in RNNs.
The study tackled the problem of limited use of modular architectures in artificial neural networks by showing that a modular recurrent neural network (RNN) outperforms a non-modular equivalent in training time, generalizability, and robustness on a memory task, using an iterative growth curriculum.
Structural modularity is a pervasive feature of biological neural networks, which have been linked to several functional and computational advantages. Yet, the use of modular architectures in artificial neural networks has been relatively limited despite early successes. Here, we explore the performance and functional dynamics of a modular network trained on a memory task via an iterative growth curriculum. We find that for a given classical, non-modular recurrent neural network (RNN), an equivalent modular network will perform better across multiple metrics, including training time, generalizability, and robustness to some perturbations. We further examine how different aspects of a modular network's connectivity contribute to its computational capability. We then demonstrate that the inductive bias introduced by the modular topology is strong enough for the network to perform well even when the connectivity within modules is fixed and only the connections between modules are trained. Our findings suggest that gradual modular growth of RNNs could provide advantages for learning increasingly complex tasks on evolutionary timescales, and help build more scalable and compressible artificial networks.