Extending Audio Context for Long-Form Understanding in Large Audio-Language Models
This work addresses a bottleneck in long-form audio understanding for LALMs, offering incremental improvements through novel methods and training strategies.
The paper tackles the problem of short audio context windows in Large Audio-Language Models (LALMs) by introducing Partial YaRN, a training-free audio-only extension method, and VLAT, a training strategy for positional augmentation, resulting in outperforming original models across settings and achieving strong performance on long audio of unseen lengths.
Large Audio-Language Models (LALMs) are often constrained by short audio context windows, even when their text backbones support long contexts, limiting long-form audio understanding. Prior work has introduced context-extension methods (e.g. YaRN) on unimodal LLMs, yet their application to LALMs remains unexplored. First, building on RoPE-based context extension, we introduce Partial YaRN, a training-free, audio-only extension method that modifies only audio token positions, leaving text positions intact to preserve the base LLM's text capabilities. Second, we propose Virtual Longform Audio Training (VLAT), a training strategy that extends Partial YaRN into a training-time positional augmentation. VLAT simulates diverse audio lengths during training, enabling generalization to inputs far longer than those seen in training and improving robustness for long-context audio understanding. Our experiments on SALMONN and Qwen2-Audio show that Partial YaRN outperforms the original models across wide range of settings, and VLAT training strategy provides substantial improvement, achieving strong performance on long audio of unseen lengths.