Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations
This work addresses the problem of training ReLU RNNs for researchers and practitioners in machine learning, offering an incremental optimization improvement.
The authors tackled the challenge of training recurrent neural networks with ReLU activations by developing a path-SGD optimization method adapted to their parameter-space geometry, resulting in significant improvements in trainability on datasets requiring long-term dependencies compared to standard SGD.
We investigate the parameter-space geometry of recurrent neural networks (RNNs), and develop an adaptation of path-SGD optimization method, attuned to this geometry, that can learn plain RNNs with ReLU activations. On several datasets that require capturing long-term dependency structure, we show that path-SGD can significantly improve trainability of ReLU RNNs compared to RNNs trained with SGD, even with various recently suggested initialization schemes.