Forget the Learning Rate, Decay Loss
This addresses the challenge of optimizing deep learning models for practitioners by potentially simplifying hyperparameter tuning, though it appears incremental as it modifies an existing optimization aspect.
The paper tackles the problem of hyperparameter tuning in deep neural network optimization by proposing a fixed learning rate with loss decay instead of adjusting the learning rate, and experiments on Image classification, Semantic segmentation, and GANs show that this strategy greatly improves model performance.
In the usual deep neural network optimization process, the learning rate is the most important hyper parameter, which greatly affects the final convergence effect. The purpose of learning rate is to control the stepsize and gradually reduce the impact of noise on the network. In this paper, we will use a fixed learning rate with method of decaying loss to control the magnitude of the update. We used Image classification, Semantic segmentation, and GANs to verify this method. Experiments show that the loss decay strategy can greatly improve the performance of the model