Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt
This work addresses the temporal perception bottleneck in Large Audio-Language Models, enabling more precise audio understanding for fine-grained tasks.
Large Audio-Language Models struggle with fine-grained temporal perception (e.g., event onset/offset). The authors propose Audio-Side Time Prompt and a Reinforcement Learning framework (TimePro-RL) to improve temporal alignment, achieving significant gains in audio grounding, sound event detection, and dense audio captioning.
Large Audio-Language Models (LALMs) enable general audio understanding and demonstrate remarkable performance across various audio tasks. However, these models still face challenges in temporal perception (e.g., inferring event onset and offset), leading to limited utility in fine-grained scenarios. To address this issue, we propose Audio-Side Time Prompt and leverage Reinforcement Learning (RL) to develop the TimePro-RL framework for fine-grained temporal perception. Specifically, we encode timestamps as embeddings and interleave them within the audio feature sequence as temporal coordinates to prompt the model. Furthermore, we introduce RL following Supervised Fine-Tuning (SFT) to directly optimize temporal alignment performance. Experiments demonstrate that TimePro-RL achieves significant performance gains across a range of audio temporal tasks, such as audio grounding, sound event detection, and dense audio captioning, validating its robust effectiveness.