Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection
This addresses the adversarial robustness problem for AVASD models, which is crucial for multi-modal applications, and is incremental as it builds on existing AVASD methods.
The paper tackles the problem of adversarial robustness in audio-visual active speaker detection (AVASD) by revealing vulnerabilities to audio-only, visual-only, and audio-visual attacks and proposing a novel audio-visual interaction loss (AVIL) for defense, which outperforms adversarial training by 33.14 mAP under multi-modal attacks.
Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications. However, to the best of our knowledge, the adversarial robustness of AVASD models hasn't been investigated, not to mention the effective defense against such attacks. In this paper, we are the first to reveal the vulnerability of AVASD models under audio-only, visual-only, and audio-visual adversarial attacks through extensive experiments. What's more, we also propose a novel audio-visual interaction loss (AVIL) for making attackers difficult to find feasible adversarial examples under an allocated attack budget. The loss aims at pushing the inter-class embeddings to be dispersed, namely non-speech and speech clusters, sufficiently disentangled, and pulling the intra-class embeddings as close as possible to keep them compact. Experimental results show the AVIL outperforms the adversarial training by 33.14 mAP (%) under multi-modal attacks.