USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding
This work addresses the need for a single, high-quality audio encoder that works across diverse domains (speech, music, general audio) for use in audio LLMs.
USAD 2.0 scales representation distillation to one billion parameters, achieving strong or state-of-the-art performance across probing and LLM-based evaluations for universal audio understanding.
Audio encoders are critical to modern audio applications as large language models (LLMs) increasingly rely on a single encoder for diverse inputs. While self-supervised learning (SSL) has yielded strong domain-specific encoders like speech or music experts, multi-domain approaches like USAD and SPEAR remain limited in coverage and evaluation. Recent studies also suggest supervised encoders align better with audio LLMs. We present USAD 2.0, a universal encoder integrating knowledge from both SSL and supervised foundation models. USAD 2.0 introduces domain-aware distillation to address teacher mismatch, extends coverage to the music domain, and adds second-stage supervised distillation for downstream use. We further scale the model to one billion parameters via depth scaling. Experiments show USAD 2.0 achieves strong or state-of-the-art performance across probing and LLM-based evaluations.