Distilling a Modular Reservoir Through a Genomic Bottleneck
This work addresses the challenge of designing efficient recurrent neural networks for temporal tasks by drawing inspiration from biological development, offering a way to generate functional network structures with minimal post-training.
The authors propose a method that uses hypernetworks to learn a compressed generative process for modular reservoir connectivity, enabling sparse recurrent networks to solve difficult temporal tasks with minimal training and high robustness.
The intricate structures of biological neural networks largely emerge during development, guided by a comparatively compressed blueprint encoded in the genome. The connectivity that emerges from this decoding process is rich in structure, and already equips the organism with functional modules upon birth. This initial structure serves as a scaffold that can be gradually refined and fine-tuned through lifelong experience, via a variety of plasticity mechanisms. Drawing inspiration from this interaction between evolutionary and developmental modes of learning, we use hypernetworks to learn a compressed generative process that generates the connectivity of a modular reservoir. We show that this marriage between curriculum-based meta-learning and modular reservoir computing can generate sparse recurrent networks that solve difficult temporal tasks with minimal training and without concessions to robustness.