Deep Learning Meets Sparse Regularization: A Signal Processing Perspective
This provides a foundational mathematical understanding for researchers in machine learning and signal processing, though it is incremental as it builds on existing techniques.
The paper tackles the lack of a rigorous mathematical theory to explain deep neural networks' performance by introducing a signal processing-based framework that characterizes their functional properties, explaining effects like weight decay regularization and sparsity in high-dimensional problems.
Deep learning has been wildly successful in practice and most state-of-the-art machine learning methods are based on neural networks. Lacking, however, is a rigorous mathematical theory that adequately explains the amazing performance of deep neural networks. In this article, we present a relatively new mathematical framework that provides the beginning of a deeper understanding of deep learning. This framework precisely characterizes the functional properties of neural networks that are trained to fit to data. The key mathematical tools which support this framework include transform-domain sparse regularization, the Radon transform of computed tomography, and approximation theory, which are all techniques deeply rooted in signal processing. This framework explains the effect of weight decay regularization in neural network training, the use of skip connections and low-rank weight matrices in network architectures, the role of sparsity in neural networks, and explains why neural networks can perform well in high-dimensional problems.