ASBench: Image Anomalies Synthesis Benchmark for Anomaly Detection
This work addresses the problem of constrained anomaly detection in manufacturing due to limited abnormal samples and high annotation costs, offering a benchmarking tool for researchers, though it is incremental as it focuses on evaluation rather than new synthesis methods.
The paper tackles the lack of systematic evaluation for anomaly synthesis methods in anomaly detection by proposing ASBench, a comprehensive benchmarking framework that assesses four key dimensions, revealing limitations in current methods and providing insights for future research.
Anomaly detection plays a pivotal role in manufacturing quality control, yet its application is constrained by limited abnormal samples and high manual annotation costs. While anomaly synthesis offers a promising solution, existing studies predominantly treat anomaly synthesis as an auxiliary component within anomaly detection frameworks, lacking systematic evaluation of anomaly synthesis algorithms. Current research also overlook crucial factors specific to anomaly synthesis, such as decoupling its impact from detection, quantitative analysis of synthetic data and adaptability across different scenarios. To address these limitations, we propose ASBench, the first comprehensive benchmarking framework dedicated to evaluating anomaly synthesis methods. Our framework introduces four critical evaluation dimensions: (i) the generalization performance across different datasets and pipelines (ii) the ratio of synthetic to real data (iii) the correlation between intrinsic metrics of synthesis images and anomaly detection performance metrics , and (iv) strategies for hybrid anomaly synthesis methods. Through extensive experiments, ASBench not only reveals limitations in current anomaly synthesis methods but also provides actionable insights for future research directions in anomaly synthesis