Distribution-Free Confidence Ellipsoids for Ridge Regression with PAC Bounds
This work addresses uncertainty quantification for practitioners in control and signal processing using regularized linear models, offering an incremental improvement with tighter theoretical bounds.
The paper tackles the problem of quantifying estimation uncertainty in ridge regression by extending the Sign-Perturbed Sums ellipsoidal outer approximation algorithm to this context, deriving probably approximately correct upper bounds for region sizes that show the effect of regularization and provide tighter bounds under weaker assumptions.
Linearly parametrized models are widely used in control and signal processing, with the least-squares (LS) estimate being the archetypical solution. When the input is insufficiently exciting, the LS problem may be unsolvable or numerically unstable. This issue can be resolved through regularization, typically with ridge regression. Although regularized estimators reduce the variance error, it remains important to quantify their estimation uncertainty. A possible approach for linear regression is to construct confidence ellipsoids with the Sign-Perturbed Sums (SPS) ellipsoidal outer approximation (EOA) algorithm. The SPS EOA builds non-asymptotic confidence ellipsoids under the assumption that the noises are independent and symmetric about zero. This paper introduces an extension of the SPS EOA algorithm to ridge regression, and derives probably approximately correct (PAC) upper bounds for the resulting region sizes. Compared with previous analyses, our result explicitly show how the regularization parameter affects the region sizes, and provide tighter bounds under weaker excitation assumptions. Finally, the practical effect of regularization is also demonstrated via simulation experiments.