Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation
It addresses the problem of segmenting sounding objects in videos for applications like video analysis, but it is incremental as it builds on existing fusion-based methods with architectural improvements.
The paper tackles the audio-visual segmentation task by proposing a novel transformer-based method to better fuse audio and visual features, resulting in improved performance and generalization in multi-sound and open-set scenarios.
The goal of the audio-visual segmentation (AVS) task is to segment the sounding objects in the video frames using audio cues. However, current fusion-based methods have the performance limitations due to the small receptive field of convolution and inadequate fusion of audio-visual features. To overcome these issues, we propose a novel \textbf{Au}dio-aware query-enhanced \textbf{TR}ansformer (AuTR) to tackle the task. Unlike existing methods, our approach introduces a multimodal transformer architecture that enables deep fusion and aggregation of audio-visual features. Furthermore, we devise an audio-aware query-enhanced transformer decoder that explicitly helps the model focus on the segmentation of the pinpointed sounding objects based on audio signals, while disregarding silent yet salient objects. Experimental results show that our method outperforms previous methods and demonstrates better generalization ability in multi-sound and open-set scenarios.