Statistical Evaluation of Anomaly Detectors for Sequences
This work addresses a methodological gap for researchers in anomaly detection, offering incremental improvements to evaluation metrics.
The paper tackled the problem of evaluating anomaly detectors for sequences by formalizing time-tolerant precision and recall measures, and demonstrated through simulation that standard measures can overestimate performance, providing a method to assess statistical significance.
Although precision and recall are standard performance measures for anomaly detection, their statistical properties in sequential detection settings are poorly understood. In this work, we formalize a notion of precision and recall with temporal tolerance for point-based anomaly detection in sequential data. These measures are based on time-tolerant confusion matrices that may be used to compute time-tolerant variants of many other standard measures. However, care has to be taken to preserve interpretability. We perform a statistical simulation study to demonstrate that precision and recall may overestimate the performance of a detector, when computed with temporal tolerance. To alleviate this problem, we show how to obtain null distributions for the two measures to assess the statistical significance of reported results.