Degrees of Freedom Analysis of Unrolled Neural Networks
It addresses the generalization mystery in unrolled networks for image restoration, providing insights for network design in low sample size regimes, but is incremental as it builds on existing SURE analysis.
This paper analyzes the generalization risk of unrolled neural networks for image restoration by using the Stein's Unbiased Risk Estimator (SURE) and degrees-of-freedom (DOF) to quantify prediction variance, showing that DOF increases with train sample size and converges to generalization risk, with recurrent networks converging faster than non-recurrent ones.
Unrolled neural networks emerged recently as an effective model for learning inverse maps appearing in image restoration tasks. However, their generalization risk (i.e., test mean-squared-error) and its link to network design and train sample size remains mysterious. Leveraging the Stein's Unbiased Risk Estimator (SURE), this paper analyzes the generalization risk with its bias and variance components for recurrent unrolled networks. We particularly investigate the degrees-of-freedom (DOF) component of SURE, trace of the end-to-end network Jacobian, to quantify the prediction variance. We prove that DOF is well-approximated by the weighted \textit{path sparsity} of the network under incoherence conditions on the trained weights. Empirically, we examine the SURE components as a function of train sample size for both recurrent and non-recurrent (with many more parameters) unrolled networks. Our key observations indicate that: 1) DOF increases with train sample size and converges to the generalization risk for both recurrent and non-recurrent schemes; 2) recurrent network converges significantly faster (with less train samples) compared with non-recurrent scheme, hence recurrence serves as a regularization for low sample size regimes.