Empirical Bayes Estimators for High-Dimensional Sparse Vectors
This work addresses estimation in high-dimensional sparse settings, which is incremental as it builds on existing methods like soft-thresholding and empirical Bayes for improved risk and loss concentration.
The paper tackles the problem of estimating high-dimensional sparse vectors in Gaussian noise by proposing an empirical Bayes estimator and a hybrid estimator that selects between it and soft-thresholding based on risk estimates, showing that the hybrid estimator's loss concentrates on the minimum of the two and its risk is close to the minimum, with simulations and applications in compressed sensing demonstrating superior performance.
The problem of estimating a high-dimensional sparse vector $\boldsymbolθ \in \mathbb{R}^n$ from an observation in i.i.d. Gaussian noise is considered. The performance is measured using squared-error loss. An empirical Bayes shrinkage estimator, derived using a Bernoulli-Gaussian prior, is analyzed and compared with the well-known soft-thresholding estimator. We obtain concentration inequalities for the Stein's unbiased risk estimate and the loss function of both estimators. The results show that for large $n$, both the risk estimate and the loss function concentrate on deterministic values close to the true risk. Depending on the underlying $\boldsymbolθ$, either the proposed empirical Bayes (eBayes) estimator or soft-thresholding may have smaller loss. We consider a hybrid estimator that attempts to pick the better of the soft-thresholding estimator and the eBayes estimator by comparing their risk estimates. It is shown that: i) the loss of the hybrid estimator concentrates on the minimum of the losses of the two competing estimators, and ii) the risk of the hybrid estimator is within order $\frac{1}{\sqrt{n}}$ of the minimum of the two risks. Simulation results are provided to support the theoretical results. Finally, we use the eBayes and hybrid estimators as denoisers in the approximate message passing (AMP) algorithm for compressed sensing, and show that their performance is superior to the soft-thresholding denoiser in a wide range of settings.