Splitting Steepest Descent for Growing Neural Architectures
This provides a new approach for learning lightweight neural architectures in resource-constrained settings, though it appears incremental as it builds on existing steepest descent methods.
The paper tackles the problem of optimizing neural network structures by adaptively growing them through neuron splitting, using a functional steepest descent approach to decide which neurons to split and update offspring, resulting in a computationally efficient method for lightweight architectures.
We develop a progressive training approach for neural networks which adaptively grows the network structure by splitting existing neurons to multiple off-springs. By leveraging a functional steepest descent idea, we derive a simple criterion for deciding the best subset of neurons to split and a splitting gradient for optimally updating the off-springs. Theoretically, our splitting strategy is a second-order functional steepest descent for escaping saddle points in an $\infty$-Wasserstein metric space, on which the standard parametric gradient descent is a first-order steepest descent. Our method provides a new computationally efficient approach for optimizing neural network structures, especially for learning lightweight neural architectures in resource-constrained settings.