Small noise analysis for Tikhonov and RKHS regularizations
This work addresses a fundamental open problem in machine learning and inverse problems by providing a theoretical framework for regularization analysis, though it is incremental in extending existing methods.
The paper tackles the comparative analysis of regularization norms in ill-posed linear inverse problems with Gaussian noise, establishing a small noise analysis framework that reveals the instability of L2-regularizers and proposes adaptive fractional RKHS regularizers to achieve optimal convergence rates, though with hyper-parameters that may decay too fast for practical selection.
Regularization plays a pivotal role in ill-posed machine learning and inverse problems. However, the fundamental comparative analysis of various regularization norms remains open. We establish a small noise analysis framework to assess the effects of norms in Tikhonov and RKHS regularizations, in the context of ill-posed linear inverse problems with Gaussian noise. This framework studies the convergence rates of regularized estimators in the small noise limit and reveals the potential instability of the conventional L2-regularizer. We solve such instability by proposing an innovative class of adaptive fractional RKHS regularizers, which covers the L2 Tikhonov and RKHS regularizations by adjusting the fractional smoothness parameter. A surprising insight is that over-smoothing via these fractional RKHSs consistently yields optimal convergence rates, but the optimal hyper-parameter may decay too fast to be selected in practice.