Sparse Polyak with optimal thresholding operators for high-dimensional M-estimation
This work addresses a bottleneck in high-dimensional statistical optimization, offering an incremental improvement for researchers and practitioners in machine learning and statistics.
The authors tackled the issue of Sparse Polyak's trade-off between convergence guarantees and solution sparsity in high-dimensional M-estimation, proposing a variant that retains scaling properties while achieving sparser and more accurate solutions.
We propose and analyze a variant of Sparse Polyak for high dimensional M-estimation problems. Sparse Polyak proposes a novel adaptive step-size rule tailored to suitably estimate the problem's curvature in the high-dimensional setting, guaranteeing that the algorithm's performance does not deteriorate when the ambient dimension increases. However, convergence guarantees can only be obtained by sacrificing solution sparsity and statistical accuracy. In this work, we introduce a variant of Sparse Polyak that retains its desirable scaling properties with respect to the ambient dimension while obtaining sparser and more accurate solutions.