Opening the Black Box of wav2vec Feature Encoder
This provides insights into self-supervised speech models for researchers, though it is incremental as it focuses on analysis rather than new methods.
The paper tackled the problem of understanding the inner workings of wav2vec's convolutional feature encoder by analyzing its latent space using synthesized audio signals, concluding that it embeds fundamental frequency, formants, amplitude, and temporal detail, and constructs a metric space for acoustic similarity.
Self-supervised models, namely, wav2vec and its variants, have shown promising results in various downstream tasks in the speech domain. However, their inner workings are poorly understood, calling for in-depth analyses on what the model learns. In this paper, we concentrate on the convolutional feature encoder where its latent space is often speculated to represent discrete acoustic units. To analyze the embedding space in a reductive manner, we feed the synthesized audio signals, which is the summation of simple sine waves. Through extensive experiments, we conclude that various information is embedded inside the feature encoder representations: (1) fundamental frequency, (2) formants, and (3) amplitude, packed with (4) sufficient temporal detail. Further, the information incorporated inside the latent representations is analogous to spectrograms but with a fundamental difference: latent representations construct a metric space so that closer representations imply acoustic similarity.