How do Minimum-Norm Shallow Denoisers Look in Function Space?
This provides theoretical insights into neural network denoisers, which are incremental but important for understanding their behavior in tasks like image reconstruction and generation.
The paper tackles the problem of theoretically characterizing shallow ReLU neural network denoisers in function space under interpolation and minimal norm conditions, deriving closed-form solutions for univariate and multivariate data and proving better generalization than empirical MMSE estimators at low noise levels.
Neural network (NN) denoisers are an essential building block in many common tasks, ranging from image reconstruction to image generation. However, the success of these models is not well understood from a theoretical perspective. In this paper, we aim to characterize the functions realized by shallow ReLU NN denoisers -- in the common theoretical setting of interpolation (i.e., zero training loss) with a minimal representation cost (i.e., minimal $\ell^2$ norm weights). First, for univariate data, we derive a closed form for the NN denoiser function, find it is contractive toward the clean data points, and prove it generalizes better than the empirical MMSE estimator at a low noise level. Next, for multivariate data, we find the NN denoiser functions in a closed form under various geometric assumptions on the training data: data contained in a low-dimensional subspace, data contained in a union of one-sided rays, or several types of simplexes. These functions decompose into a sum of simple rank-one piecewise linear interpolations aligned with edges and/or faces connecting training samples. We empirically verify this alignment phenomenon on synthetic data and real images.