Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs
This work addresses efficiency and scalability in speech processing for researchers and practitioners, though it is incremental as it builds on existing neural audio codec trends.
The paper tackles the problem of speech representation learning by introducing Codec2Vec, a framework using discrete audio codec units, which achieves competitive performance on the SUPERB benchmark while reducing storage by up to 16.5x and training time by 2.3x.
Recent advancements in neural audio codecs have not only enabled superior audio compression but also enhanced speech synthesis techniques. Researchers are now exploring their potential as universal acoustic feature extractors for a broader range of speech processing tasks. Building on this trend, we introduce Codec2Vec, the first speech representation learning framework that relies exclusively on discrete audio codec units. This approach offers several advantages, including improved data storage and transmission efficiency, faster training, and enhanced data privacy. We explore masked prediction with various training target derivation strategies to thoroughly understand the effectiveness of this framework. Evaluated on the SUPERB benchmark, Codec2Vec achieves competitive performance compared to continuous-input models while reducing storage requirements by up to 16.5x and training time by 2.3x, showcasing its scalability and efficiency.