On Pixel-level Performance Assessment in Anomaly Detection
This work addresses a methodological problem for researchers and practitioners in anomaly detection, but it is incremental as it focuses on improving evaluation metrics rather than detection methods.
The paper tackled the challenge of assessing pixel-level performance in anomaly detection due to class imbalance, concluding that Precision-Recall-based metrics better capture relative method performance based on experiments with eleven methods on twenty-one problems.
Anomaly detection methods have demonstrated remarkable success across various applications. However, assessing their performance, particularly at the pixel-level, presents a complex challenge due to the severe imbalance that is most commonly present between normal and abnormal samples. Commonly adopted evaluation metrics designed for pixel-level detection may not effectively capture the nuanced performance variations arising from this class imbalance. In this paper, we dissect the intricacies of this challenge, underscored by visual evidence and statistical analysis, leading to delve into the need for evaluation metrics that account for the imbalance. We offer insights into more accurate metrics, using eleven leading contemporary anomaly detection methods on twenty-one anomaly detection problems. Overall, from this extensive experimental evaluation, we can conclude that Precision-Recall-based metrics can better capture relative method performance, making them more suitable for the task.