Compressive Sensing and Neural Networks from a Statistical Learning Perspective
This work addresses the lack of theoretical foundations for unfolded neural networks in compressive sensing, offering a generalization analysis that could benefit researchers in signal processing and machine learning, though it is incremental as it builds on existing iterative methods.
The authors tackled the problem of providing theoretical guarantees for neural networks used in sparse signal reconstruction from compressive measurements, presenting a generalization error bound that scales logarithmically with network depth and linearly with the number of measurements.
Various iterative reconstruction algorithms for inverse problems can be unfolded as neural networks. Empirically, this approach has often led to improved results, but theoretical guarantees are still scarce. While some progress on generalization properties of neural networks have been made, great challenges remain. In this chapter, we discuss and combine these topics to present a generalization error analysis for a class of neural networks suitable for sparse reconstruction from few linear measurements. The hypothesis class considered is inspired by the classical iterative soft-thresholding algorithm (ISTA). The neural networks in this class are obtained by unfolding iterations of ISTA and learning some of the weights. Based on training samples, we aim at learning the optimal network parameters via empirical risk minimization and thereby the optimal network that reconstructs signals from their compressive linear measurements. In particular, we may learn a sparsity basis that is shared by all of the iterations/layers and thereby obtain a new approach for dictionary learning. For this class of networks, we present a generalization bound, which is based on bounding the Rademacher complexity of hypothesis classes consisting of such deep networks via Dudley's integral. Remarkably, under realistic conditions, the generalization error scales only logarithmically in the number of layers, and at most linear in number of measurements.