Progressive Learning for Systematic Design of Large Neural Networks
This work addresses the challenge of manual network design and hyperparameter tuning for researchers and practitioners in machine learning, though it appears incremental as it builds on existing activation functions and optimization methods.
The authors tackled the problem of systematically designing large neural networks by introducing a progressive learning algorithm that optimizes layers sequentially using convex optimization, achieving good generalization in classification and regression tasks on standard databases.
We develop an algorithm for systematic design of a large artificial neural network using a progression property. We find that some non-linear functions, such as the rectifier linear unit and its derivatives, hold the property. The systematic design addresses the choice of network size and regularization of parameters. The number of nodes and layers in network increases in progression with the objective of consistently reducing an appropriate cost. Each layer is optimized at a time, where appropriate parameters are learned using convex optimization. Regularization parameters for convex optimization do not need a significant manual effort for tuning. We also use random instances for some weight matrices, and that helps to reduce the number of parameters we learn. The developed network is expected to show good generalization power due to appropriate regularization and use of random weights in the layers. This expectation is verified by extensive experiments for classification and regression problems, using standard databases.