Parameter Averaging for Feature Ranking
This work addresses the issue of unreliable feature ranking for users of neural networks in tabular data analysis, though it is incremental as it builds on existing parameter averaging techniques.
The paper tackles the problem of neural network sensitivity to initialization in feature ranking by introducing XTab, a parameter averaging method that improves robustness and accuracy in identifying ground-truth feature importance, as demonstrated through experiments on synthetic and real-world data.
Neural Networks are known to be sensitive to initialisation. The methods that rely on neural networks for feature ranking are not robust since they can have variations in their ranking when the model is initialized and trained with different random seeds. In this work, we introduce a novel method based on parameter averaging to estimate accurate and robust feature importance in tabular data setting, referred as XTab. We first initialize and train multiple instances of a shallow network (referred as local masks) with "different random seeds" for a downstream task. We then obtain a global mask model by "averaging the parameters" of local masks. We show that although the parameter averaging might result in a global model with higher loss, it still leads to the discovery of the ground-truth feature importance more consistently than an individual model does. We conduct extensive experiments on a variety of synthetic and real-world data, demonstrating that the XTab can be used to obtain the global feature importance that is not sensitive to sub-optimal model initialisation.