A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning
This addresses efficiency bottlenecks for deploying audio models on resource-constrained devices, representing an incremental improvement over existing methods.
The paper tackles the high computational cost and parameterization of self-supervised audio representation learning by proposing HEAR, a decoupled architecture inspired by human cognition, which achieves competitive performance with only 15M parameters and 9.47 GFLOPs.
While self-supervised learning (SSL) has revolutionized audio representation, the excessive parameterization and quadratic computational cost of standard Transformers limit their deployment on resource-constrained devices. To address this bottleneck, we propose HEAR (Human-inspired Efficient Audio Representation), a novel decoupled architecture. Inspired by the human cognitive ability to isolate local acoustic features from global context, HEAR splits the processing pipeline into two dedicated modules: an Acoustic Model for local feature extraction and a Task Model for global semantic integration. Coupled with an Acoustic Tokenizer trained via knowledge distillation, our approach enables robust Masked Audio Modeling (MAM). Extensive experiments demonstrate that HEAR requires only 15M parameters and 9.47 GFLOPs for inference, operating at a fraction of the computational cost of conventional foundation models (which typically require 85M-94M parameters). Despite this high efficiency, HEAR achieves highly competitive performance across diverse audio classification benchmarks. The code and pre-trained models are available at https://github.com/HarunoriKawano/HEAR