FIB: A Method for Evaluation of Feature Impact Balance in Multi-Dimensional Data
This addresses the need for better error analysis in machine learning by providing a tool to assess feature impact balance, though it is incremental as it builds on existing evaluation methods.
The paper tackles the problem of imbalance in feature contributions to errors in multi-dimensional data by proposing the Feature Impact Balance (FIB) score, which quantifies this balance on a scale from 0 to 1, and demonstrates its application in model selection for various tasks using AutoEncoders and Variational AutoEncoders.
Errors might not have the same consequences depending on the task at hand. Nevertheless, there is limited research investigating the impact of imbalance in the contribution of different features in an error vector. Therefore, we propose the Feature Impact Balance (FIB) score. It measures whether there is a balanced impact of features in the discrepancies between two vectors. We designed the FIB score to lie in [0, 1]. Scores close to 0 indicate that a small number of features contribute to most of the error, and scores close to 1 indicate that most features contribute to the error equally. We experimentally study the FIB on different datasets, using AutoEncoders and Variational AutoEncoders. We show how the feature impact balance varies during training and showcase its usability to support model selection for single output and multi-output tasks.