Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video
This addresses the challenge of audio-visual integration for applications like assistive technologies, though it is incremental as it builds on existing memory network approaches.
The paper tackles the problem of generating audio from silent video by introducing a multi-modal bridging framework that uses a memory network to store and relate visual and audio representations, enabling speech reconstruction and lip reading with only visual input, achieving state-of-the-art performance.
In this paper, we introduce a novel audio-visual multi-modal bridging framework that can utilize both audio and visual information, even with uni-modal inputs. We exploit a memory network that stores source (i.e., visual) and target (i.e., audio) modal representations, where source modal representation is what we are given, and target modal representations are what we want to obtain from the memory network. We then construct an associative bridge between source and target memories that considers the interrelationship between the two memories. By learning the interrelationship through the associative bridge, the proposed bridging framework is able to obtain the target modal representations inside the memory network, even with the source modal input only, and it provides rich information for its downstream tasks. We apply the proposed framework to two tasks: lip reading and speech reconstruction from silent video. Through the proposed associative bridge and modality-specific memories, each task knowledge is enriched with the recalled audio context, achieving state-of-the-art performance. We also verify that the associative bridge properly relates the source and target memories.