You Only Speak Once to See
This work addresses the underexplored use of audio for object recognition and grounding, potentially improving robotic systems and computer vision applications, though it appears incremental as it builds on existing pre-trained models and methods.
The paper tackles the problem of grounding objects in images using audio cues, introducing YOSS to map speech commands to objects via contrastive learning and multi-modal alignment, with experimental results showing that audio guidance can effectively enhance object grounding.
Grounding objects in images using visual cues is a well-established approach in computer vision, yet the potential of audio as a modality for object recognition and grounding remains underexplored. We introduce YOSS, "You Only Speak Once to See," to leverage audio for grounding objects in visual scenes, termed Audio Grounding. By integrating pre-trained audio models with visual models using contrastive learning and multi-modal alignment, our approach captures speech commands or descriptions and maps them directly to corresponding objects within images. Experimental results indicate that audio guidance can be effectively applied to object grounding, suggesting that incorporating audio guidance may enhance the precision and robustness of current object grounding methods and improve the performance of robotic systems and computer vision applications. This finding opens new possibilities for advanced object recognition, scene understanding, and the development of more intuitive and capable robotic systems.