Local Evaluation of Time Series Anomaly Detection Algorithms
This work addresses the need for more reliable and interpretable evaluation metrics in time series anomaly detection, which is crucial for researchers and practitioners in fields like monitoring and fault detection, though it is incremental as it builds on existing precision/recall frameworks.
The authors tackled the problem of evaluating time series anomaly detection algorithms by highlighting limitations in existing metrics and proposing a new theoretically grounded, parameter-free extension to precision and recall based on 'affiliation' between ground truth and predictions, which is robust against adversary strategies and enables fine-grained interpretation.
In recent years, specific evaluation metrics for time series anomaly detection algorithms have been developed to handle the limitations of the classical precision and recall. However, such metrics are heuristically built as an aggregate of multiple desirable aspects, introduce parameters and wipe out the interpretability of the output. In this article, we first highlight the limitations of the classical precision/recall, as well as the main issues of the recent event-based metrics -- for instance, we show that an adversary algorithm can reach high precision and recall on almost any dataset under weak assumption. To cope with the above problems, we propose a theoretically grounded, robust, parameter-free and interpretable extension to precision/recall metrics, based on the concept of ``affiliation'' between the ground truth and the prediction sets. Our metrics leverage measures of duration between ground truth and predictions, and have thus an intuitive interpretation. By further comparison against random sampling, we obtain a normalized precision/recall, quantifying how much a given set of results is better than a random baseline prediction. By construction, our approach keeps the evaluation local regarding ground truth events, enabling fine-grained visualization and interpretation of algorithmic results. We compare our proposal against various public time series anomaly detection datasets, algorithms and metrics. We further derive theoretical properties of the affiliation metrics that give explicit expectations about their behavior and ensure robustness against adversary strategies.