LALR: Theoretical and Experimental validation of Lipschitz Adaptive Learning Rate in Regression and Neural Networks
This work addresses convergence speed issues in regression and neural networks, presenting a novel theoretical framework with significant experimental gains.
The authors tackled the problem of slow convergence in regression tasks by proposing a Lipschitz-based adaptive learning rate policy for Mean Absolute Error and Quantile loss functions, achieving up to 20x faster convergence compared to constant learning rates.
We propose a theoretical framework for an adaptive learning rate policy for the Mean Absolute Error loss function and Quantile loss function and evaluate its effectiveness for regression tasks. The framework is based on the theory of Lipschitz continuity, specifically utilizing the relationship between learning rate and Lipschitz constant of the loss function. Based on experimentation, we have found that the adaptive learning rate policy enables up to 20x faster convergence compared to a constant learning rate policy.