Using theoretical ROC curves for analysing machine learning binary classifiers
This work addresses the need for better theoretical analysis of classifier performance in machine learning, particularly for applications like biometric security, but it is incremental as it builds on existing ROC curve methods.
The paper tackles the problem of analyzing binary classifiers by proposing the estimation of theoretical probability distributions for classifier responses, which is typically omitted in machine learning. In a biometric security example, they fit beta distributions to logistic regression and ANN classifier responses to categorize extremal behaviors at the ends of ROC curves.
Most binary classifiers work by processing the input to produce a scalar response and comparing it to a threshold value. The various measures of classifier performance assume, explicitly or implicitly, probability distributions $P_s$ and $P_n$ of the response belonging to either class, probability distributions for the cost of each type of misclassification, and compute a performance score from the expected cost. In machine learning, classifier responses are obtained experimentally and performance scores are computed directly from them, without any assumptions on $P_s$ and $P_n$. Here, we argue that the omitted step of estimating theoretical distributions for $P_s$ and $P_n$ can be useful. In a biometric security example, we fit beta distributions to the responses of two classifiers, one based on logistic regression and one on ANNs, and use them to establish a categorisation into a small number of classes with different extremal behaviours at the ends of the ROC curves.