Speech-Based Visual Question Answering
This addresses the challenge of multimodal AI integration for accessibility or hands-free applications, but is incremental as it adapts existing VQA techniques to speech input.
The paper tackles the problem of visual question answering using spoken questions, introducing two methods: an end-to-end neural network using audio waveforms and a pipelined approach with ASR followed by text-based VQA, and finds both methods tolerate noise at similar levels.
This paper introduces speech-based visual question answering (VQA), the task of generating an answer given an image and a spoken question. Two methods are studied: an end-to-end, deep neural network that directly uses audio waveforms as input versus a pipelined approach that performs ASR (Automatic Speech Recognition) on the question, followed by text-based visual question answering. Furthermore, we investigate the robustness of both methods by injecting various levels of noise into the spoken question and find both methods to be tolerate noise at similar levels.