Benchmarking Unsupervised Outlier Detection with Realistic Synthetic Data
This work addresses the problem of evaluating outlier detection methods for researchers, but it is incremental as it builds on existing synthetic data approaches.
The authors tackled the difficulty of benchmarking unsupervised outlier detection by proposing a generic process for generating realistic synthetic data, which reconstructs regular instances from real-world data and creates outliers with specific characteristics, and they demonstrated its practicality through a benchmark with state-of-the-art methods.
Benchmarking unsupervised outlier detection is difficult. Outliers are rare, and existing benchmark data contains outliers with various and unknown characteristics. Fully synthetic data usually consists of outliers and regular instance with clear characteristics and thus allows for a more meaningful evaluation of detection methods in principle. Nonetheless, there have only been few attempts to include synthetic data in benchmarks for outlier detection. This might be due to the imprecise notion of outliers or to the difficulty to arrive at a good coverage of different domains with synthetic data. In this work we propose a generic process for the generation of data sets for such benchmarking. The core idea is to reconstruct regular instances from existing real-world benchmark data while generating outliers so that they exhibit insightful characteristics. This allows both for a good coverage of domains and for helpful interpretations of results. We also describe three instantiations of the generic process that generate outliers with specific characteristics, like local outliers. A benchmark with state-of-the-art detection methods confirms that our generic process is indeed practical.