Revisiting Multimodal Fusion for 3D Anomaly Detection from an Architectural Perspective
This work addresses a gap in 3D anomaly detection by focusing on architectural design, which is incremental as it builds on existing fusion strategies and NAS paradigms.
The paper tackles the problem of multimodal fusion architecture design for 3D anomaly detection by systematically analyzing its impact and proposing 3D-ADNAS, a neural architecture search method that searches across fusion strategies and modality-specific modules, resulting in consistent improvements in accuracy, frame rate, and memory usage across various model capacities.
Existing efforts to boost multimodal fusion of 3D anomaly detection (3D-AD) primarily concentrate on devising more effective multimodal fusion strategies. However, little attention was devoted to analyzing the role of multimodal fusion architecture (topology) design in contributing to 3D-AD. In this paper, we aim to bridge this gap and present a systematic study on the impact of multimodal fusion architecture design on 3D-AD. This work considers the multimodal fusion architecture design at the intra-module fusion level, i.e., independent modality-specific modules, involving early, middle or late multimodal features with specific fusion operations, and also at the inter-module fusion level, i.e., the strategies to fuse those modules. In both cases, we first derive insights through theoretically and experimentally exploring how architectural designs influence 3D-AD. Then, we extend SOTA neural architecture search (NAS) paradigm and propose 3D-ADNAS to simultaneously search across multimodal fusion strategies and modality-specific modules for the first time.Extensive experiments show that 3D-ADNAS obtains consistent improvements in 3D-AD across various model capacities in terms of accuracy, frame rate, and memory usage, and it exhibits great potential in dealing with few-shot 3D-AD tasks.