Contextual Feature Selection with Conditional Stochastic Gates
This addresses the need for context-aware feature selection in machine learning applications across various domains, representing a novel method rather than an incremental improvement.
The paper tackles the problem of feature selection when feature relevance varies with context, proposing Conditional Stochastic Gates (c-STG) to conditionally select features based on context variables, resulting in improved feature selection capabilities and prediction accuracy across simulated and real-world datasets.
Feature selection is a crucial tool in machine learning and is widely applied across various scientific disciplines. Traditional supervised methods generally identify a universal set of informative features for the entire population. However, feature relevance often varies with context, while the context itself may not directly affect the outcome variable. Here, we propose a novel architecture for contextual feature selection where the subset of selected features is conditioned on the value of context variables. Our new approach, Conditional Stochastic Gates (c-STG), models the importance of features using conditional Bernoulli variables whose parameters are predicted based on contextual variables. We introduce a hypernetwork that maps context variables to feature selection parameters to learn the context-dependent gates along with a prediction model. We further present a theoretical analysis of our model, indicating that it can improve performance and flexibility over population-level methods in complex feature selection settings. Finally, we conduct an extensive benchmark using simulated and real-world datasets across multiple domains demonstrating that c-STG can lead to improved feature selection capabilities while enhancing prediction accuracy and interpretability.