Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?
This work addresses the problem of improving speech recognition accuracy in noisy environments for applications like automated transcription, though it is incremental as it builds on existing multimodal approaches.
The paper investigates how multiple modalities (audio, image context, lip information) affect speech recognition accuracy, finding that integrating more modalities can increase accuracy, with images providing the greatest benefit at moderate noise levels and performance improving when relevant visual information is filtered.
Decoder-only discrete-token language models have recently achieved significant success in automatic speech recognition. However, systematic analyses of how different modalities impact performance in specific scenarios remain limited. In this paper, we investigate the effects of multiple modalities on recognition accuracy on both synthetic and real-world datasets. Our experiments suggest that: (1) Integrating more modalities can increase accuracy; in particular, our paper is, to our best knowledge, the first to show the benefit of combining audio, image context, and lip information; (2) Images as a supplementary modality for speech recognition provide the greatest benefit at moderate noise levels, moreover, they exhibit a different trend compared to inherently synchronized modalities like lip movements; (3) Performance improves on both synthetic and real-world datasets when the most relevant visual information is filtered as a preprocessing step.