Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources
This work addresses the gap between audio-language models (which lack spatial reasoning) and sound event localization models (which lack semantic coverage), enabling joint spatio-temporal reasoning for dynamic sound sources.
The authors introduce ST-AudioQA, a spatio-temporal audio QA dataset with FOA renderings, and propose ST-AudioLM, a model that jointly learns event semantics and source trajectories. Their method improves the semantic-localization tradeoff and achieves stronger reasoning performance than static spatial and localization-oriented baselines.
Sound events are entities with semantic identities, locations, and trajectories, but current audio-language models usually reason about clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To address this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and questions about what is sounding, where it is, how it moves, and how sources relate. We further propose ST-Audio Encoder, a time-resolved FOA audio encoder that learns event semantics together with source trajectories, and ST-AudioLM, which connects the audio tokens from the encoder to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the semantic-localization tradeoff and yields stronger reasoning performance than static spatial and localization-oriented baselines.