AutoLR: An Evolutionary Approach to Learning Rate Policies
This work addresses the need for tailored learning rate optimization in specific network topologies, though it is incremental as it builds on existing automatic methods.
The authors tackled the problem of optimizing learning rate policies for specific neural network architectures by introducing AutoLR, a framework that evolves learning rate schedulers using Structured Grammatical Evolution. Results showed that certain evolved policies were more efficient than a common baseline, suggesting this approach can improve neural network performance.
The choice of a proper learning rate is paramount for good Artificial Neural Network training and performance. In the past, one had to rely on experience and trial-and-error to find an adequate learning rate. Presently, a plethora of state of the art automatic methods exist that make the search for a good learning rate easier. While these techniques are effective and have yielded good results over the years, they are general solutions. This means the optimization of learning rate for specific network topologies remains largely unexplored. This work presents AutoLR, a framework that evolves Learning Rate Schedulers for a specific Neural Network Architecture using Structured Grammatical Evolution. The system was used to evolve learning rate policies that were compared with a commonly used baseline value for learning rate. Results show that training performed using certain evolved policies is more efficient than the established baseline and suggest that this approach is a viable means of improving a neural network's performance.