Object Referring in Visual Scene with Spoken Language
This work addresses object referring for human-machine interaction by using more natural spoken language instead of text, though it is incremental as it builds on existing text-based methods.
The paper tackles object referring with spoken language by introducing two datasets and a novel approach that decomposes the problem into three sub-problems with task-specific vision-language interactions, resulting in consistent and significant performance improvements over competing methods, including robustness to audio noise.
Object referring has important applications, especially for human-machine interaction. While having received great attention, the task is mainly attacked with written language (text) as input rather than spoken language (speech), which is more natural. This paper investigates Object Referring with Spoken Language (ORSpoken) by presenting two datasets and one novel approach. Objects are annotated with their locations in images, text descriptions and speech descriptions. This makes the datasets ideal for multi-modality learning. The approach is developed by carefully taking down ORSpoken problem into three sub-problems and introducing task-specific vision-language interactions at the corresponding levels. Experiments show that our method outperforms competing methods consistently and significantly. The approach is also evaluated in the presence of audio noise, showing the efficacy of the proposed vision-language interaction methods in counteracting background noise.