Interpretable Anomaly Detection with DIFFI: Depth-based Isolation Forest Feature Importance
This work addresses interpretability issues in anomaly detection for applications like system monitoring, though it is incremental as it builds on the existing Isolation Forest method.
The paper tackles the lack of interpretability in Isolation Forest for anomaly detection by proposing DIFFI, a method to compute feature importance scores globally and locally, and includes a procedure for unsupervised feature selection. The results are assessed on synthetic and real-world datasets with comparisons to state-of-the-art techniques, and the code is made publicly available.
Anomaly Detection is an unsupervised learning task aimed at detecting anomalous behaviours with respect to historical data. In particular, multivariate Anomaly Detection has an important role in many applications thanks to the capability of summarizing the status of a complex system or observed phenomenon with a single indicator (typically called `Anomaly Score') and thanks to the unsupervised nature of the task that does not require human tagging. The Isolation Forest is one of the most commonly adopted algorithms in the field of Anomaly Detection, due to its proven effectiveness and low computational complexity. A major problem affecting Isolation Forest is represented by the lack of interpretability, an effect of the inherent randomness governing the splits performed by the Isolation Trees, the building blocks of the Isolation Forest. In this paper we propose effective, yet computationally inexpensive, methods to define feature importance scores at both global and local level for the Isolation Forest. Moreover, we define a procedure to perform unsupervised feature selection for Anomaly Detection problems based on our interpretability method; such procedure also serves the purpose of tackling the challenging task of feature importance evaluation in unsupervised anomaly detection. We assess the performance on several synthetic and real-world datasets, including comparisons against state-of-the-art interpretability techniques, and make the code publicly available to enhance reproducibility and foster research in the field.