Multiclass ROC
This work addresses the need for robust evaluation metrics in multi-class classification, particularly for statisticians and machine learning practitioners dealing with imbalanced data and misclassification costs, though it is incremental as it builds on existing binary ROC/AUC concepts.
The paper tackled the problem of evaluating multi-class classifiers by generalizing ROC/AUC analysis, addressing issues like lack of sensible plots and sensitivity to imbalanced data, and achieved this through a method that provides a one-dimensional vector representation and binary AUC-equivalent summary.
Model evaluation is of crucial importance in modern statistics application. The construction of ROC and calculation of AUC have been widely used for binary classification evaluation. Recent research generalizing the ROC/AUC analysis to multi-class classification has problems in at least one of the four areas: 1. failure to provide sensible plots 2. being sensitive to imbalanced data 3. unable to specify mis-classification cost and 4. unable to provide evaluation uncertainty quantification. Borrowing from a binomial matrix factorization model, we provide an evaluation metric summarizing the pair-wise multi-class True Positive Rate (TPR) and False Positive Rate (FPR) with one-dimensional vector representation. Visualization on the representation vector measures the relative speed of increment between TPR and FPR across all the classes pairs, which in turns provides a ROC plot for the multi-class counterpart. An integration over those factorized vector provides a binary AUC-equivalent summary on the classifier performance. Mis-clasification weights specification and bootstrapped confidence interval are also enabled to accommodate a variety of of evaluation criteria. To support our findings, we conducted extensive simulation studies and compared our method to the pair-wise averaged AUC statistics on benchmark datasets.