Calibrated Principal Component Regression
This work addresses bias in statistical inference for overparameterized models, which is an incremental improvement for researchers in machine learning and statistics.
The paper tackles the truncation bias problem in Principal Component Regression (PCR) for overparameterized generalized linear models by proposing Calibrated Principal Component Regression (CPCR), which learns a prior in the principal component subspace and calibrates it in the original feature space, resulting in improved prediction across multiple overparameterized problems.
We propose a new method for statistical inference in generalized linear models. In the overparameterized regime, Principal Component Regression (PCR) reduces variance by projecting high-dimensional data to a low-dimensional principal subspace before fitting. However, PCR incurs truncation bias whenever the true regression vector has mass outside the retained principal components (PC). To mitigate the bias, we propose Calibrated Principal Component Regression (CPCR), which first learns a low-variance prior in the PC subspace and then calibrates the model in the original feature space via a centered Tikhonov step. CPCR leverages cross-fitting and controls the truncation bias by softening PCR's hard cutoff. Theoretically, we calculate the out-of-sample risk in the random matrix regime, which shows that CPCR outperforms standard PCR when the regression signal has non-negligible components in low-variance directions. Empirically, CPCR consistently improves prediction across multiple overparameterized problems. The results highlight CPCR's stability and flexibility in modern overparameterized settings.