Principled Architecture-aware Scaling of Hyperparameters
This work addresses the challenge of hyperparameter tuning for researchers and practitioners in deep learning, offering an incremental improvement by incorporating architecture-aware scaling.
The authors tackled the problem of hyperparameter selection for deep neural networks by characterizing how initializations and maximal learning rates depend on network architecture, enabling generalization across MLPs and CNNs with complex topologies. They demonstrated that their approach can alter network rankings in benchmarks, highlighting the need for fair comparisons in AutoML.
Training a high-quality deep neural network requires choosing suitable hyperparameters, which is a non-trivial and expensive process. Current works try to automatically optimize or design principles of hyperparameters, such that they can generalize to diverse unseen scenarios. However, most designs or optimization methods are agnostic to the choice of network structures, and thus largely ignore the impact of neural architectures on hyperparameters. In this work, we precisely characterize the dependence of initializations and maximal learning rates on the network architecture, which includes the network depth, width, convolutional kernel size, and connectivity patterns. By pursuing every parameter to be maximally updated with the same mean squared change in pre-activations, we can generalize our initialization and learning rates across MLPs (multi-layer perception) and CNNs (convolutional neural network) with sophisticated graph topologies. We verify our principles with comprehensive experiments. More importantly, our strategy further sheds light on advancing current benchmarks for architecture design. A fair comparison of AutoML algorithms requires accurate network rankings. However, we demonstrate that network rankings can be easily changed by better training networks in benchmarks with our architecture-aware learning rates and initialization.