Unsupervised Speech Representation Pooling Using Vector Quantization
This addresses the pooling bottleneck for applying general-purpose speech models to multiple downstream tasks, offering an incremental improvement over existing methods.
The paper tackles the problem of pooling variable-length speech representations from self-supervised models by proposing a novel unsupervised method using vector quantization, which achieves competitive performance across tasks like keyword spotting and emotion recognition without additional training.
With the advent of general-purpose speech representations from large-scale self-supervised models, applying a single model to multiple downstream tasks is becoming a de-facto approach. However, the pooling problem remains; the length of speech representations is inherently variable. The naive average pooling is often used, even though it ignores the characteristics of speech, such as differently lengthed phonemes. Hence, we design a novel pooling method to squash acoustically similar representations via vector quantization, which does not require additional training, unlike attention-based pooling. Further, we evaluate various unsupervised pooling methods on various self-supervised models. We gather diverse methods scattered around speech and text to evaluate on various tasks: keyword spotting, speaker identification, intent classification, and emotion recognition. Finally, we quantitatively and qualitatively analyze our method, comparing it with supervised pooling methods.