DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Models
This work addresses the challenge of integrating spatial audio into AI systems for applications like robotics or AR/VR, representing an incremental advancement in audio processing methods.
The paper tackles the problem of spatial audio reasoning with large language models by introducing DSpAST, a novel audio encoder that learns disentangled representations for sound events, direction, and distance, achieving significant performance improvements over SpatialAST on the SpatialSoundQA benchmark.
Reasoning about spatial audio with large language models requires a spatial audio encoder as an acoustic front-end to obtain audio embeddings for further processing. Such an encoder needs to capture all information required to detect the type of sound events, as well as the direction and distance of their corresponding sources. Accomplishing this with a single audio encoder is demanding as the information required for each of these tasks is mostly independent of each other. As a result, the performance obtained with a single encoder is often worse than when using task-specific audio encoders. In this work, we present DSpAST, a novel audio encoder based on SpatialAST that learns disentangled representations of spatial audio while having only 0.2% additional parameters. Experiments on SpatialSoundQA with the spatial audio reasoning system BAT demonstrate that DSpAST significantly outperforms SpatialAST.