Slow Kill for Big Data Learning
This addresses variable selection problems for researchers and practitioners dealing with large-scale data, though it appears incremental as it builds on existing optimization and statistical methods.
The paper tackles the challenge of variable selection and parameter estimation in big-data applications by introducing a novel technique called 'slow kill', which combines nonconvex constrained optimization, adaptive shrinkage, and increasing learning rates, and it outperforms state-of-the-art algorithms in experiments on real and synthetic data while being computationally efficient.
Big-data applications often involve a vast number of observations and features, creating new challenges for variable selection and parameter estimation. This paper presents a novel technique called ``slow kill,'' which utilizes nonconvex constrained optimization, adaptive $\ell_2$-shrinkage, and increasing learning rates. The fact that the problem size can decrease during the slow kill iterations makes it particularly effective for large-scale variable screening. The interaction between statistics and optimization provides valuable insights into controlling quantiles, stepsize, and shrinkage parameters in order to relax the regularity conditions required to achieve the desired level of statistical accuracy. Experimental results on real and synthetic data show that slow kill outperforms state-of-the-art algorithms in various situations while being computationally efficient for large-scale data.