Escaping the Procrustean Bed: Groupwise Orthogonal Connectors for Audio-Language Models
Addresses a fundamental compression failure in audio-language model connectors, enabling better preservation of paralinguistic cues for downstream tasks.
Audio-language models suffer from output collapse in the Q-Former connector, losing paralinguistic information. ORCA splits queries into groups with orthogonal constraints, achieving 75.2% on SAKURA multi-hop reasoning (26.4 points over baseline) and reducing query redundancy by 12x while increasing cross-speaker variance by 75x.
Audio-language models compress a speech encoder's output through a Querying Transformer (Q-Former) connector before feeding it to a large language model. We identify two failures in this compression. The connector's output vectors collapse to a single direction, and different speakers produce nearly indistinguishable outputs, with paralinguistic cues such as speaker identity, gender, and prosody lost along the way. Our method, ORCA, reverses this collapse by splitting the queries into groups whose outputs are constrained to point in different directions. On SAKURA multi-hop reasoning, ORCA gains 26.4 points over an identically trained 4B baseline, reaching 75.2% (vs. 49.0% for the 8B Audio Flamingo-3). At the connector level, the same change cuts query redundancy by 12x and raises cross-speaker variance by 75x.