Evaluation of Deep Audio Representations for Hearables
This work addresses the problem of effectively steering hearable devices, which is significant for users of such devices, and presents an incremental improvement in the field of audio representation models.
The authors tackled the problem of evaluating deep audio representations for hearables and found that the BEATs model significantly outperforms its counterparts, with the DEAR dataset and benchmark providing a comprehensive evaluation framework. The dataset consists of 1,158 audio tracks, each 30 seconds long, and encompasses eight tasks to assess various acoustic properties.
Effectively steering hearable devices requires understanding the acoustic environment around the user. In the computational analysis of sound scenes, foundation models have emerged as the state of the art to produce high-performance, robust, multi-purpose audio representations. We introduce and release Deep Evaluation of Audio Representations (DEAR), the first dataset and benchmark to evaluate the efficacy of foundation models in capturing essential acoustic properties for hearables. The dataset includes 1,158 audio tracks, each 30 seconds long, created by spatially mixing proprietary monologues with commercial, high-quality recordings of everyday acoustic scenes. Our benchmark encompasses eight tasks that assess the general context, speech sources, and technical acoustic properties of the audio scenes. Through our evaluation of four general-purpose audio representation models, we demonstrate that the BEATs model significantly surpasses its counterparts. This superiority underscores the advantage of models trained on diverse audio collections, confirming their applicability to a wide array of auditory tasks, including encoding the environment properties necessary for hearable steering. The DEAR dataset and associated code are available at https://dear-dataset.github.io.