Tune smarter not harder: A principled approach to tuning learning rates for shallow nets
This work addresses the challenge of efficient hyperparameter tuning for practitioners in machine learning, offering a method that reduces computational cost while improving performance, though it is incremental as it builds on existing tuning approaches.
The authors tackled the problem of hyperparameter tuning for shallow neural networks by proposing a principled method to choose learning rates based on the gradient Lipschitz constant, which significantly outperformed existing methods like Tree Parzen Estimators in simulations across applications such as channel estimation and currency rate prediction.
Effective hyper-parameter tuning is essential to guarantee the performance that neural networks have come to be known for. In this work, a principled approach to choosing the learning rate is proposed for shallow feedforward neural networks. We associate the learning rate with the gradient Lipschitz constant of the objective to be minimized while training. An upper bound on the mentioned constant is derived and a search algorithm, which always results in non-divergent traces, is proposed to exploit the derived bound. It is shown through simulations that the proposed search method significantly outperforms the existing tuning methods such as Tree Parzen Estimators (TPE). The proposed method is applied to three different existing applications: a) channel estimation in OFDM systems, b) prediction of the exchange currency rates and c) offset estimation in OFDM receivers, and it is shown to pick better learning rates than the existing methods using the same or lesser compute power.