AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models
This addresses the underexplored trustworthiness risks in ALLMs for secure deployment in real-world audio scenarios, though it is incremental as it builds on existing evaluation frameworks by adapting them to audio-specific vulnerabilities.
The authors tackled the problem of evaluating trustworthiness in Audio Large Language Models (ALLMs) by proposing AudioTrust, a comprehensive framework that assesses six dimensions across 26 sub-tasks using over 4,420 audio samples, revealing significant limitations in 14 state-of-the-art models.
Audio Large Language Models (ALLMs) have gained widespread adoption, yet their trustworthiness remains underexplored. Existing evaluation frameworks, designed primarily for text, fail to address unique vulnerabilities introduced by audio's acoustic properties. We identify significant trustworthiness risks in ALLMs arising from non-semantic acoustic cues, including timbre, accent, and background noise, which can manipulate model behavior. We propose AudioTrust, a comprehensive framework for systematic evaluation of ALLM trustworthiness across audio-specific risks. AudioTrust encompasses six key dimensions: fairness, hallucination, safety, privacy, robustness, and authentication. The framework implements 26 distinct sub-tasks using a curated dataset of over 4,420 audio samples from real-world scenarios, including daily conversations, emergency calls, and voice assistant interactions. We conduct comprehensive evaluations across 18 experimental configurations using human-validated automated pipelines. Our evaluation of 14 state-of-the-art open-source and closed-source ALLMs reveals significant limitations when confronted with diverse high-risk audio scenarios, providing insights for secure deployment of audio models. Code and data are available at https://github.com/JusperLee/AudioTrust.