Learning Visual Affordance from Audio
This addresses the challenge of affordance grounding in robotics and AI by leveraging audio cues to overcome limitations like ambiguity or occlusion in existing methods, though it is incremental as it builds on related audio-visual tasks.
The paper tackles the problem of segmenting object interaction regions from action sounds, introducing the Audio-Visual Affordance Grounding (AV-AG) task and a corresponding dataset, with the proposed AVAGFormer model achieving state-of-the-art performance.
We introduce Audio-Visual Affordance Grounding (AV-AG), a new task that segments object interaction regions from action sounds. Unlike existing approaches that rely on textual instructions or demonstration videos, which often limited by ambiguity or occlusion, audio provides real-time, semantically rich, and visually independent cues for affordance grounding, enabling more intuitive understanding of interaction regions. To support this task, we construct the first AV-AG dataset, comprising a large collection of action sounds, object images, and pixel-level affordance annotations. The dataset also includes an unseen subset to evaluate zero-shot generalization. Furthermore, we propose AVAGFormer, a model equipped with a semantic-conditioned cross-modal mixer and a dual-head decoder that effectively fuses audio and visual signals for mask prediction. Experiments show that AVAGFormer achieves state-of-the-art performance on AV-AG, surpassing baselines from related tasks. Comprehensive analyses highlight the distinctions between AV-AG and AVS, the benefits of end-to-end modeling, and the contribution of each component. Code and dataset have been released on https://jscslld.github.io/AVAGFormer/.