Modularity as a Means for Complexity Management in Neural Networks Learning
This addresses the problem of optimization complexity in neural networks for researchers and practitioners, but it is incremental as it builds on existing modular concepts.
The authors tackled the complexity of training large neural networks by proposing a modular design that decomposes the network into control and functional modules, and demonstrated improved training speed, stability, and maintainability on a list sorting problem.
Training a Neural Network (NN) with lots of parameters or intricate architectures creates undesired phenomena that complicate the optimization process. To address this issue we propose a first modular approach to NN design, wherein the NN is decomposed into a control module and several functional modules, implementing primitive operations. We illustrate the modular concept by comparing performances between a monolithic and a modular NN on a list sorting problem and show the benefits in terms of training speed, training stability and maintainability. We also discuss some questions that arise in modular NNs.