Population-based Global Optimisation Methods for Learning Long-term Dependencies with RNNs
This addresses the problem of training RNNs for time-series forecasting, but it is incremental as it applies existing optimization techniques to a known bottleneck.
The paper tackled the difficulty of training RNNs to learn long-term dependencies by using gradient-free population-based optimization methods, showing that evolution strategies and particle swarm optimization improved performance in volatility forecasting, with ES being the most consistent across architectures.
Despite recent innovations in network architectures and loss functions, training RNNs to learn long-term dependencies remains difficult due to challenges with gradient-based optimisation methods. Inspired by the success of Deep Neuroevolution in reinforcement learning (Such et al. 2017), we explore the use of gradient-free population-based global optimisation (PBO) techniques -- training RNNs to capture long-term dependencies in time-series data. Testing evolution strategies (ES) and particle swarm optimisation (PSO) on an application in volatility forecasting, we demonstrate that PBO methods lead to performance improvements in general, with ES exhibiting the most consistent results across a variety of architectures.