Minimax and Communication-Efficient Distributed Best Subset Selection with Oracle Property
This addresses the need for distributed statistical inference with true sparsity in fields like finance and e-commerce, representing a significant advance in distributed sparse estimation.
The authors tackled the problem of distributed best subset selection in high-dimensional datasets by proposing a two-stage algorithm that efficiently estimates the active set and refines sparse estimates, achieving the minimax ℓ₂ error bound and greatly reducing communication costs.
The explosion of large-scale data in fields such as finance, e-commerce, and social media has outstripped the processing capabilities of single-machine systems, driving the need for distributed statistical inference methods. Traditional approaches to distributed inference often struggle with achieving true sparsity in high-dimensional datasets and involve high computational costs. We propose a novel, two-stage, distributed best subset selection algorithm to address these issues. Our approach starts by efficiently estimating the active set while adhering to the $\ell_0$ norm-constrained surrogate likelihood function, effectively reducing dimensionality and isolating key variables. A refined estimation within the active set follows, ensuring sparse estimates and matching the minimax $\ell_2$ error bound. We introduce a new splicing technique for adaptive parameter selection to tackle subproblems under $\ell_0$ constraints and a Generalized Information Criterion (GIC). Our theoretical and numerical studies show that the proposed algorithm correctly finds the true sparsity pattern, has the oracle property, and greatly lowers communication costs. This is a big step forward in distributed sparse estimation.