Estimator of Prediction Error Based on Approximate Message Passing for Penalized Linear Regression
This work addresses the challenge of accurate prediction error estimation in statistical modeling for researchers and practitioners, particularly in sparse regression contexts, but it is incremental as it builds on existing AMP and Stein's lemma frameworks.
The authors tackled the problem of estimating prediction error for penalized linear regression, especially with sparse penalties, by proposing an estimator based on approximate message passing (AMP) that is asymptotically unbiased under Gaussian assumptions and performs well with real data under nonconvex penalties, selecting models close to minimizing true prediction error.
We propose an estimator of prediction error using an approximate message passing (AMP) algorithm that can be applied to a broad range of sparse penalties. Following Stein's lemma, the estimator of the generalized degrees of freedom, which is a key quantity for the construction of the estimator of the prediction error, is calculated at the AMP fixed point. The resulting form of the AMP-based estimator does not depend on the penalty function, and its value can be further improved by considering the correlation between predictors. The proposed estimator is asymptotically unbiased when the components of the predictors and response variables are independently generated according to a Gaussian distribution. We examine the behaviour of the estimator for real data under nonconvex sparse penalties, where Akaike's information criterion does not correspond to an unbiased estimator of the prediction error. The model selected by the proposed estimator is close to that which minimizes the true prediction error.