A classification performance evaluation measure considering data separability
This work addresses the problem of biased performance evaluation in machine learning for researchers and practitioners by providing a more comprehensive measure, though it is incremental as it supplements existing methods.
The paper tackles the bias in evaluating classification models by proposing a new data separability measure called the rate of separability (RS), based on data coding rate, and validates it on synthetic datasets and a real multi-task scenario, showing positive correlation with recognition accuracy.
Machine learning and deep learning classification models are data-driven, and the model and the data jointly determine their classification performance. It is biased to evaluate the model's performance only based on the classifier accuracy while ignoring the data separability. Sometimes, the model exhibits excellent accuracy, which might be attributed to its testing on highly separable data. Most of the current studies on data separability measures are defined based on the distance between sample points, but this has been demonstrated to fail in several circumstances. In this paper, we propose a new separability measure--the rate of separability (RS), which is based on the data coding rate. We validate its effectiveness as a supplement to the separability measure by comparing it to four other distance-based measures on synthetic datasets. Then, we demonstrate the positive correlation between the proposed measure and recognition accuracy in a multi-task scenario constructed from a real dataset. Finally, we discuss the methods for evaluating the classification performance of machine learning and deep learning models considering data separability.