A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training of DNNs
This addresses the need for automated and efficient training optimization in deep learning, particularly for adversarial training, though it appears incremental as it builds on existing tuning methods.
The paper tackles the problem of manually tuning learning rates for training neural networks, especially in adversarial scenarios or with new models, by proposing a parameterless automated algorithm that consistently achieves top-level accuracy across datasets and models.
Training neural networks on image datasets generally require extensive experimentation to find the optimal learning rate regime. Especially, for the cases of adversarial training or for training a newly synthesized model, one would not know the best learning rate regime beforehand. We propose an automated algorithm for determining the learning rate trajectory, that works across datasets and models for both natural and adversarial training, without requiring any dataset/model specific tuning. It is a stand-alone, parameterless, adaptive approach with no computational overhead. We theoretically discuss the algorithm's convergence behavior. We empirically validate our algorithm extensively. Our results show that our proposed approach \emph{consistently} achieves top-level accuracy compared to SOTA baselines in the literature in natural as well as adversarial training.