SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Models
For researchers developing LALMs, this work offers a data-efficient method to enhance auditory understanding by targeting specific perceptual weaknesses with synthetic signals.
The paper identifies fine-grained spectrotemporal perceptual weaknesses in large audio language models (LALMs) and proposes SpectCount, a data-efficient fine-tuning method using fully synthetic audio signals. This approach resolves the weaknesses and improves performance across diverse auditory benchmarks (sound, music, speech) without relying on real-world data.
Large audio language models (LALMs) extend large language models with an audio encoder and large-scale audio data. However, the scarcity of high-quality annotated audio data remains a fundamental bottleneck for scaling. Through probing signal detectability analysis, we identify fine-grained spectrotemporal perceptual weaknesses in a foundation LALM. To address these challenges, we propose Spectrotemporal Counting (SpectCount), a data-efficient fine-tuning approach based on fully synthetic audio signals generated on-the-fly, without relying on real-world audio, annotations, or pretrained generative models. SpectCount not only resolves the observed weaknesses but also improves performance on diverse auditory benchmarks spanning sound, music, and speech, unseen during fine-tuning. These results suggest that weakness-targeted synthetic signals provide a data-efficient path toward enhanced auditory understanding capabilities in LALMs.