GradMax: Growing Neural Networks using Gradient Information
This addresses the challenge for machine learning practitioners of efficiently growing neural networks without retraining, though it is incremental as it builds on existing architecture optimization methods.
The paper tackles the problem of costly retraining when modifying neural network architectures by introducing GradMax, a method that adds new neurons during training without disrupting learned parameters, improving training dynamics through gradient maximization and SVD-based initialization, and demonstrates effectiveness in various vision tasks and architectures.
The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified. In this work we instead focus on growing the architecture without requiring costly retraining. We present a method that adds new neurons during training without impacting what is already learned, while improving the training dynamics. We achieve the latter by maximizing the gradients of the new weights and find the optimal initialization efficiently by means of the singular value decomposition (SVD). We call this technique Gradient Maximizing Growth (GradMax) and demonstrate its effectiveness in variety of vision tasks and architectures.