SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition
This addresses training inefficiencies in deep CNNs for speech recognition, offering a more efficient alternative to existing methods.
The paper tackled the difficulty of training very deep CNNs for speech recognition by proposing SNDCNN, which replaces shortcut connections and batch normalization with SELU activations, achieving up to 4.5% lower word error rate and 60%-80% faster training and inference speeds compared to ResNet-50.
Very deep CNNs achieve state-of-the-art results in both computer vision and speech recognition, but are difficult to train. The most popular way to train very deep CNNs is to use shortcut connections (SC) together with batch normalization (BN). Inspired by Self- Normalizing Neural Networks, we propose the self-normalizing deep CNN (SNDCNN) based acoustic model topology, by removing the SC/BN and replacing the typical RELU activations with scaled exponential linear unit (SELU) in ResNet-50. SELU activations make the network self-normalizing and remove the need for both shortcut connections and batch normalization. Compared to ResNet- 50, we can achieve the same or lower (up to 4.5% relative) word error rate (WER) while boosting both training and inference speed by 60%-80%. We also explore other model inference optimization schemes to further reduce latency for production use.