SpeechIQ: Speech Intelligence Quotient Across Cognitive Levels in Voice Understanding Large Language Models
This provides a novel, human cognition-inspired evaluation method for voice understanding models, addressing the need for more comprehensive assessment beyond traditional metrics, though it is incremental in applying cognitive principles to a specific domain.
The paper introduces Speech-based Intelligence Quotient (SIQ), a new evaluation pipeline for voice understanding large language models, which assesses abilities across three cognitive levels and quantifies performance with metrics like word error rate and QA accuracy. It demonstrates that SIQ enables unified comparisons between models, identifies benchmark errors, and detects hallucinations.
We introduce Speech-based Intelligence Quotient (SIQ) as a new form of human cognition-inspired evaluation pipeline for voice understanding large language models, LLM Voice, designed to assess their voice understanding ability. Moving beyond popular voice understanding metrics such as word error rate (WER), SIQ examines LLM Voice across three cognitive levels motivated by Bloom's Taxonomy: (1) Remembering (i.e., WER for verbatim accuracy); (2) Understanding (i.e., similarity of LLM's interpretations); and (3) Application (i.e., QA accuracy for simulating downstream tasks). We demonstrate that SIQ not only quantifies voice understanding abilities but also provides unified comparisons between cascaded methods (e.g., ASR LLM) and end-to-end models, identifies annotation errors in existing benchmarks, and detects hallucinations in LLM Voice. Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks, while exposing overlooked challenges in multi-modal training.