Preservation of Language Understanding Capabilities in Speech-aware Large Language Models
This addresses the problem of ensuring speech interfaces do not degrade performance for users, but it is incremental as it focuses on benchmarking rather than solving the underlying issue.
The paper introduces C3T, a benchmark to test how well speech-aware large language models maintain language understanding when accessed through speech, quantifying fairness across speaker categories and robustness across modalities.
The paper presents C3T (Cross-modal Capabilities Conservation Test), a new benchmark for assessing the performance of speech-aware large language models. The benchmark utilizes textual tasks and a voice cloning text-to-speech model to quantify the extent to which language understanding capabilities are preserved when the model is accessed via speech input. C3T quantifies the fairness of the model for different categories of speakers and its robustness across text and speech modalities.