Joint Audio and Speech Understanding
This addresses the need for machines to understand complex audio environments like humans, though it appears incremental as it combines existing modules.
The authors tackled the problem of joint understanding of speech and non-speech audio by building LTU-AS, a model that integrates Whisper and LLaMA to simultaneously recognize and comprehend spoken text, speech paralinguistics, and non-speech audio events, achieving a universal audio perception capability.
Humans are surrounded by audio signals that include both speech and non-speech sounds. The recognition and understanding of speech and non-speech audio events, along with a profound comprehension of the relationship between them, constitute fundamental cognitive capabilities. For the first time, we build a machine learning model, called LTU-AS, that has a conceptually similar universal audio perception and advanced reasoning ability. Specifically, by integrating Whisper as a perception module and LLaMA as a reasoning module, LTU-AS can simultaneously recognize and jointly understand spoken text, speech paralinguistics, and non-speech audio events - almost everything perceivable from audio signals.