Splines-Based Feature Importance in Kolmogorov-Arnold Networks: A Framework for Supervised Tabular Data Dimensionality Reduction
This provides an interpretable alternative for feature selection in supervised tabular data analysis, though it is incremental as it builds on existing KAN frameworks.
The paper tackled feature selection for high-dimensional tabular data by proposing splines-based methods using Kolmogorov-Arnold networks (KANs), resulting in KAN-based selectors that are competitive or superior to classical baselines, with average F1 and R² scores showing robust performance across datasets.
High-dimensional datasets require effective feature selection to improve predictive performance, interpretability, and robustness. We propose and evaluate feature selection methods for tabular datasets based on Kolmogorov-Arnold networks (KANs), which parameterize feature transformations through splines, enabling direct access to interpretable importance measures. We introduce four KAN-based selectors ($\textit{KAN-L1}$, $\textit{KAN-L2}$, $\textit{KAN-SI}$, $\textit{KAN-KO}$) and compare them against classical baselines (LASSO, Random Forest, Mutual Information, SVM-RFE) across multiple classification and regression tabular dataset benchmarks. Average (over three retention levels: 20\%, 40\%, and 60\%) F1 scores and $R^2$ score results reveal that KAN-based selectors, particularly $\textit{KAN-L2}$, $\textit{KAN-L1}$, $\textit{KAN-SI}$, and $\textit{KAN-KO}$, are competitive with and sometimes superior to classical baselines in structured and synthetic datasets. However, $\textit{KAN-L1}$ is often too aggressive in regression, removing useful features, while $\textit{KAN-L2}$ underperforms in classification, where simple coefficient shrinkage misses complex feature interactions. $\textit{KAN-L2}$ and $\textit{KAN-SI}$ provide robust performance on noisy regression datasets and heterogeneous datasets, aligning closely with ensemble predictors. In classification tasks, KAN selectors such as $\textit{KAN-L1}$, $\textit{KAN-KO}$, and $\textit{KAN-SI}$ sometimes surpass the other selectors by eliminating redundancy, particularly in high-dimensional multi-class data. Overall, our findings demonstrate that KAN-based feature selection provides a powerful and interpretable alternative to traditional methods, capable of uncovering nonlinear and multivariate feature relevance beyond sparsity or impurity-based measures.