Robustness assessment of large audio language models in multiple-choice evaluation
This work addresses the problem of unreliable evaluation metrics for audio language models, which is incremental as it builds on existing MCQA frameworks.
The study investigated the robustness of large audio language models in multiple-choice question answering, finding that models are sensitive to changes like choice ordering and paraphrasing, and proposed a new evaluation protocol to account for these variations.
Recent advances in large audio language models (LALMs) have primarily been assessed using a multiple-choice question answering (MCQA) framework. However, subtle changes, such as shifting the order of choices, result in substantially different results. Existing MCQA frameworks do not account for this variability and report a single accuracy number per benchmark or category. We dive into the MCQA evaluation framework and conduct a systematic study spanning three benchmarks (MMAU, MMAR and MMSU) and four models: Audio Flamingo 2, Audio Flamingo 3, Qwen2.5-Omni-7B-Instruct, and Kimi-Audio-7B-Instruct. Our findings indicate that models are sensitive not only to the ordering of choices, but also to the paraphrasing of the question and the choices. Finally, we propose a simpler evaluation protocol and metric that account for subtle variations and provide a more detailed evaluation report of LALMs within the MCQA framework.