Comparison of Anomaly Detectors: Context Matters
This work addresses the problem of inconsistent and contradictory results in anomaly detection research, providing clarity for researchers and practitioners by identifying key contextual factors that affect method performance.
This paper compares various anomaly detection methods, including deep generative models, on tabular and image datasets to identify sources of variability in their performance. The study found that experimental conditions, such as dataset type, anomaly nature, and hyperparameter selection strategy (especially the number of anomalies in the validation set), significantly influence which method performs best.
Deep generative models are challenging the classical methods in the field of anomaly detection nowadays. Every new method provides evidence of outperforming its predecessors, often with contradictory results. The objective of this comparison is twofold: to compare anomaly detection methods of various paradigms with focus on deep generative models, and identification of sources of variability that can yield different results. The methods were compared on popular tabular and image datasets. We identified the main sources of variability to be experimental conditions: i) the type data set (tabular or image) and the nature of anomalies (statistical or semantic), and ii) strategy of selection of hyperparameters, especially the number of available anomalies in the validation set. Different methods perform the best in different contexts, i.e. combination of experimental conditions together with computational time. This explains the variability of the previous results and highlights the importance of careful specification of the context in the publication of a new method. All our code and results are available for download.