Training Very Deep Networks
This addresses a key bottleneck in deep learning for researchers and practitioners by making very deep networks trainable, though it is an incremental improvement building on LSTM concepts.
The paper tackles the difficulty of training very deep neural networks by introducing highway networks, which enable direct training with simple gradient descent even for hundreds of layers, facilitating the study of extremely deep architectures.
Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.