The MVTec AD 2 Dataset: Advanced Scenarios for Unsupervised Anomaly Detection
This addresses the need for more discriminatory benchmarks in unsupervised anomaly detection for industrial inspection, though it is incremental as it builds on prior datasets.
The authors tackled the saturation of performance on existing anomaly detection benchmarks by introducing MVTec AD 2, a dataset with over 8000 high-resolution images featuring challenging industrial scenarios, and showed that state-of-the-art methods achieve below 60% average AU-PRO on it.
In recent years, performance on existing anomaly detection benchmarks like MVTec AD and VisA has started to saturate in terms of segmentation AU-PRO, with state-of-the-art models often competing in the range of less than one percentage point. This lack of discriminatory power prevents a meaningful comparison of models and thus hinders progress of the field, especially when considering the inherent stochastic nature of machine learning results. We present MVTec AD 2, a collection of eight anomaly detection scenarios with more than 8000 high-resolution images. It comprises challenging and highly relevant industrial inspection use cases that have not been considered in previous datasets, including transparent and overlapping objects, dark-field and back light illumination, objects with high variance in the normal data, and extremely small defects. We provide comprehensive evaluations of state-of-the-art methods and show that their performance remains below 60% average AU-PRO. Additionally, our dataset provides test scenarios with lighting condition changes to assess the robustness of methods under real-world distribution shifts. We host a publicly accessible evaluation server that holds the pixel-precise ground truth of the test set (https://benchmark.mvtec.com/). All image data is available at https://www.mvtec.com/company/research/datasets/mvtec-ad-2.