Bayesian feature selection with strongly-regularizing priors maps to the Ising Model
This work addresses the computational difficulty of feature selection for researchers and practitioners dealing with datasets where features outnumber samples, offering a novel theoretical mapping that simplifies the process.
The paper tackles the problem of feature selection in high-dimensional datasets by showing that Bayesian inference with strongly-regularizing priors reduces to calculating magnetizations of an Ising model, enabling explicit expressions for generalized linear models. It demonstrates this approach on the notMNIST dataset for logistic regression, achieving improved feature selection accuracy.
Identifying small subsets of features that are relevant for prediction and/or classification tasks is a central problem in machine learning and statistics. The feature selection task is especially important, and computationally difficult, for modern datasets where the number of features can be comparable to, or even exceed, the number of samples. Here, we show that feature selection with Bayesian inference takes a universal form and reduces to calculating the magnetizations of an Ising model, under some mild conditions. Our results exploit the observation that the evidence takes a universal form for strongly-regularizing priors --- priors that have a large effect on the posterior probability even in the infinite data limit. We derive explicit expressions for feature selection for generalized linear models, a large class of statistical techniques that include linear and logistic regression. We illustrate the power of our approach by analyzing feature selection in a logistic regression-based classifier trained to distinguish between the letters B and D in the notMNIST dataset.