SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
This addresses the challenge of manual learning rate tuning for deep learning practitioners, offering an automated method to enhance model performance, though it appears incremental as it builds on existing sharpness-aware optimization concepts.
The paper tackled the problem of improving generalization in deep learning by automating learning rate scheduling, proposing SALR, a sharpness-aware technique that dynamically adjusts learning rates based on local loss sharpness; results showed improved generalization, faster convergence, and solutions in flatter regions.
In an effort to improve generalization in deep learning and automate the process of learning rate scheduling, we propose SALR: a sharpness-aware learning rate update technique designed to recover flat minimizers. Our method dynamically updates the learning rate of gradient-based optimizers based on the local sharpness of the loss function. This allows optimizers to automatically increase learning rates at sharp valleys to increase the chance of escaping them. We demonstrate the effectiveness of SALR when adopted by various algorithms over a broad range of networks. Our experiments indicate that SALR improves generalization, converges faster, and drives solutions to significantly flatter regions.