Navigating the Metric Maze: A Taxonomy of Evaluation Metrics for Anomaly Detection in Time Series
This work addresses the problem of inconsistent metric selection for researchers and practitioners in time series anomaly detection, but it is incremental as it organizes existing metrics rather than introducing new ones.
The paper tackles the challenge of selecting appropriate evaluation metrics for time series anomaly detection by analyzing twenty metrics and defining a taxonomy based on their calculation methods, arguing that metric choice must be tailored to specific task requirements.
The field of time series anomaly detection is constantly advancing, with several methods available, making it a challenge to determine the most appropriate method for a specific domain. The evaluation of these methods is facilitated by the use of metrics, which vary widely in their properties. Despite the existence of new evaluation metrics, there is limited agreement on which metrics are best suited for specific scenarios and domain, and the most commonly used metrics have faced criticism in the literature. This paper provides a comprehensive overview of the metrics used for the evaluation of time series anomaly detection methods, and also defines a taxonomy of these based on how they are calculated. By defining a set of properties for evaluation metrics and a set of specific case studies and experiments, twenty metrics are analyzed and discussed in detail, highlighting the unique suitability of each for specific tasks. Through extensive experimentation and analysis, this paper argues that the choice of evaluation metric must be made with care, taking into account the specific requirements of the task at hand.