Deep Unitary Convolutional Neural Networks
This addresses training stability and efficiency issues for deep learning practitioners, though it is incremental as it builds on prior unitary methods.
The paper tackled the exploding and vanishing activation problem in deep neural networks by extending the unitary framework to networks of any dimensionality, achieving up to 32% faster inference speeds and 50% reduction in disk space while maintaining competitive accuracy.
Deep neural networks can suffer from the exploding and vanishing activation problem, in which the networks fail to train properly because the neural signals either amplify or attenuate across the layers and become saturated. While other normalization methods aim to fix the stated problem, most of them have inference speed penalties in those applications that require running averages of the neural activations. Here we extend the unitary framework based on Lie algebra to neural networks of any dimensionalities, overcoming the major constraints of the prior arts that limit synaptic weights to be square matrices. Our proposed unitary convolutional neural networks deliver up to 32% faster inference speeds and up to 50% reduction in permanent hard disk space while maintaining competitive prediction accuracy.