Classifier comparison using precision
This work addresses a gap in literature for researchers needing robust statistical comparisons of classifiers based on precision, though it appears incremental as a survey and extension of existing methods.
The paper tackles the problem of comparing classifiers using precision instead of accuracy, presenting statistical methods that account for inter-precision correlation and extend to multi-class and cross-validation scenarios, with applications to deep architectures.
New proposed models are often compared to state-of-the-art using statistical significance testing. Literature is scarce for classifier comparison using metrics other than accuracy. We present a survey of statistical methods that can be used for classifier comparison using precision, accounting for inter-precision correlation arising from use of same dataset. Comparisons are made using per-class precision and methods presented to test global null hypothesis of an overall model comparison. Comparisons are extended to multiple multi-class classifiers and to models using cross validation or its variants. Partial Bayesian update to precision is introduced when population prevalence of a class is known. Applications to compare deep architectures are studied.