Structuring Relevant Feature Sets with Multiple Model Learning
This work addresses the need for structured feature selection in scientific knowledge discovery, offering a tool to improve understanding of feature relationships, though it appears incremental in advancing existing feature selection methods.
The paper tackles the problem of uncovering structures within relevant feature sets, such as non-replaceable and functionally similar features, to enhance interpretability in high-dimensional datasets. It proposes a new learning paradigm and algorithm, demonstrating that the learned models achieve high predictive agreement and similar performance to models using the full feature set.
Feature selection is one of the most prominent learning tasks, especially in high-dimensional datasets in which the goal is to understand the mechanisms that underly the learning dataset. However most of them typically deliver just a flat set of relevant features and provide no further information on what kind of structures, e.g. feature groupings, might underly the set of relevant features. In this paper we propose a new learning paradigm in which our goal is to uncover the structures that underly the set of relevant features for a given learning problem. We uncover two types of features sets, non-replaceable features that contain important information about the target variable and cannot be replaced by other features, and functionally similar features sets that can be used interchangeably in learned models, given the presence of the non-replaceable features, with no change in the predictive performance. To do so we propose a new learning algorithm that learns a number of disjoint models using a model disjointness regularization constraint together with a constraint on the predictive agreement of the disjoint models. We explore the behavior of our approach on a number of high-dimensional datasets, and show that, as expected by their construction, these satisfy a number of properties. Namely, model disjointness, a high predictive agreement, and a similar predictive performance to models learned on the full set of relevant features. The ability to structure the set of relevant features in such a manner can become a valuable tool in different applications of scientific knowledge discovery.