LALM-as-a-Judge: Benchmarking Large Audio-Language Models for Safety Evaluation in Multi-Turn Spoken Dialogues
For researchers and practitioners evaluating safety in spoken dialogues, this benchmark and analysis reveal the limitations and trade-offs of current audio-language models in detecting socially harmful content.
The paper introduces LALM-as-a-Judge, a benchmark of 24,000 multi-turn spoken dialogues with unsafe content, and evaluates 6 large audio-language models as judges across text-only, audio-only, and multimodal setups. Results show audio provides non-lexical evidence beyond text, but multimodal gains vary and are limited by audio pathway bottlenecks.
Evaluation of socially unsafe content in spoken dialogues remains text-centric, missing prosody and transcription failures. We present LALM-as-a-Judge, which includes an open benchmark of 24,000 multi-turn spoken dialogues with one localized unsafe turn, generated out of 8 socially unsafe categories and 5 severity levels. We evaluate 6 large audio-language models (LALMs) as judges, open and closed-source, in text-only, audio-only, and multimodal setups by their sensitivity, severity-order specificity, and turn-position bias for socially harmful content in the dialogue. Results show that audio contributes non-lexical evidence beyond transcript semantics and that multimodal gains are not universal but can be text-anchored, balanced, conservative, and interfering, which we link to the audio pathway bottlenecks and fusion limits. We position the benchmark as diagnostic and derive practitioner guidance for model, modality, and prompts choices.