Is AUC the best measure for practical comparison of anomaly detectors?
This work addresses the evaluation of anomaly detection methods for practitioners, highlighting limitations in standard metrics and proposing more practical alternatives.
The paper questions whether AUC is the best metric for comparing anomaly detectors, finding that variations emphasizing low false positive rates better align with practitioner needs, but also reveals that representative anomalous samples are often required, suggesting a shift toward active or few-shot learning.
The area under receiver operating characteristics (AUC) is the standard measure for comparison of anomaly detectors. Its advantage is in providing a scalar number that allows a natural ordering and is independent on a threshold, which allows to postpone the choice. In this work, we question whether AUC is a good metric for anomaly detection, or if it gives a false sense of comfort, due to relying on assumptions which are unlikely to hold in practice. Our investigation shows that variations of AUC emphasizing accuracy at low false positive rate seem to be better correlated with the needs of practitioners, but also that we can compare anomaly detectors only in the case when we have representative examples of anomalous samples. This last result is disturbing, as it suggests that in many cases, we should do active or few-show learning instead of pure anomaly detection.