Spatial Audio Motion Understanding and Reasoning
This work addresses the challenge of spatial audio reasoning for machines, particularly in understanding moving sources, which is incremental as it builds on existing audio and language models.
The paper tackles the problem of interpreting dynamic auditory scenes by detecting overlapping events and estimating their spatial attributes, then using a large language model to answer complex queries about moving sources, achieving performance improvements over a baseline on a new benchmark dataset.
Spatial audio reasoning enables machines to interpret auditory scenes by understanding events and their spatial attributes. In this work, we focus on spatial audio understanding with an emphasis on reasoning about moving sources. First, we introduce a spatial audio encoder that processes spatial audio to detect multiple overlapping events and estimate their spatial attributes, Direction of Arrival (DoA) and source distance, at the frame level. To generalize to unseen events, we incorporate an audio grounding model that aligns audio features with semantic audio class text embeddings via a cross-attention mechanism. Second, to answer complex queries about dynamic audio scenes involving moving sources, we condition a large language model (LLM) on structured spatial attributes extracted by our model. Finally, we introduce a spatial audio motion understanding and reasoning benchmark dataset and demonstrate our framework's performance against the baseline model.