Visualization of Tradeoff in Evaluation: from Precision-Recall & PN to LIFT, ROC & BIRD
This work addresses visualization challenges in evaluation metrics for researchers and practitioners dealing with multiclass and unbalanced data, but it is incremental as it builds on existing methods.
The paper tackles the problem of evaluating classification systems with trade-offs and multiclass data by reviewing existing graphical methods like Precision-Recall and ROC, and introduces new probabilistic and information-theoretic variants of LIFT charts to better handle unbalanced and multiclass scenarios.
Evaluation often aims to reduce the correctness or error characteristics of a system down to a single number, but that always involves trade-offs. Another way of dealing with this is to quote two numbers, such as Recall and Precision, or Sensitivity and Specificity. But it can also be useful to see more than this, and a graphical approach can explore sensitivity to cost, prevalence, bias, noise, parameters and hyper-parameters. Moreover, most techniques are implicitly based on two balanced classes, and our ability to visualize graphically is intrinsically two dimensional, but we often want to visualize in a multiclass context. We review the dichotomous approaches relating to Precision, Recall, and ROC as well as the related LIFT chart, exploring how they handle unbalanced and multiclass data, and deriving new probabilistic and information theoretic variants of LIFT that help deal with the issues associated with the handling of multiple and unbalanced classes.