Enhancing XR Auditory Realism via Multimodal Scene-Aware Acoustic Rendering
This addresses the issue of auditory-visual mismatch for XR users, enhancing immersion, though it appears incremental as it builds on existing acoustic rendering methods.
The paper tackled the problem of real-time adaptation of spatial audio rendering to diverse physical scenes in Extended Reality (XR) to reduce sensory mismatch, and the result was the introduction of SAMOSA, a system that dynamically adapts to environments and synthesizes realistic Room Impulse Responses, validated through technical and expert evaluations.
In Extended Reality (XR), rendering sound that accurately simulates real-world acoustics is pivotal in creating lifelike and believable virtual experiences. However, existing XR spatial audio rendering methods often struggle with real-time adaptation to diverse physical scenes, causing a sensory mismatch between visual and auditory cues that disrupts user immersion. To address this, we introduce SAMOSA, a novel on-device system that renders spatially accurate sound by dynamically adapting to its physical environment. SAMOSA leverages a synergistic multimodal scene representation by fusing real-time estimations of room geometry, surface materials, and semantic-driven acoustic context. This rich representation then enables efficient acoustic calibration via scene priors, allowing the system to synthesize a highly realistic Room Impulse Response (RIR). We validate our system through technical evaluation using acoustic metrics for RIR synthesis across various room configurations and sound types, alongside an expert evaluation (N=12). Evaluation results demonstrate SAMOSA's feasibility and efficacy in enhancing XR auditory realism.