MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR
This addresses the challenge of complex acoustic environments for XR users, offering an incremental improvement by integrating audio-visual cues for sound separation.
The paper tackles the problem of entangled sound sources in Extended Reality (XR) environments, which overwhelm users and reduce scene awareness and social engagement, by introducing MoXaRt, a real-time system that separates up to 5 concurrent sources with ~2 second latency, resulting in a 36.2% increase in speech intelligibility and reduced cognitive load.
In Extended Reality (XR), complex acoustic environments often overwhelm users, compromising both scene awareness and social engagement due to entangled sound sources. We introduce MoXaRt, a real-time XR system that uses audio-visual cues to separate these sources and enable fine-grained sound interaction. MoXaRt's core is a cascaded architecture that performs coarse, audio-only separation in parallel with visual detection of sources (e.g., faces, instruments). These visual anchors then guide refinement networks to isolate individual sources, separating complex mixes of up to 5 concurrent sources (e.g., 2 voices + 3 instruments) with ~2 second processing latency. We validate MoXaRt through a technical evaluation on a new dataset of 30 one-minute recordings featuring concurrent speech and music, and a 22-participant user study. Empirical results indicate that our system significantly enhances speech intelligibility, yielding a 36.2% (p < 0.01) increase in listening comprehension within adversarial acoustic environments while substantially reducing cognitive load (p < 0.001), thereby paving the way for more perceptive and socially adept XR experiences.