Enhancing BEST-RQ Pseudo-Label Quality through Online Refinement for Automatic Speech Recognition
This work provides incremental improvements to self-supervised speech representation learning for ASR, specifically enhancing the BEST-RQ method's performance.
The authors improved BEST-RQ's pseudo-label quality for ASR by replacing the quantizer's linear projection with PCA, adding iterative codebook refinement, and introducing codebook distillation. On LibriSpeech test-other, they achieved a 12% relative WER reduction (10.1% to 8.8%) with 100 hours of fine-tuning data.
BEST-RQ is a simple and effective self-supervised training method for speech representation learning that performs well on automatic speech recognition (ASR) tasks. It generates pseudolabels using a fixed online quantization scheme, which simplifies training but provides weaker supervision than HuBERT-style models that iteratively refine pseudo-labels. In this work, we improve online pseudo-label generation while preserving simplicity. We propose three modifications: replacing the quantizer's linear projection with Principal Component Analysis (PCA), updating the codebook via iterative codebook refinement, and introducing an additional codebook updated via codebook distillation. We pre-train on the LibriSpeech 960-hour dataset and fine-tune using 100 hours of supervised LibriSpeech data. With all three modifications enabled, we achieve a 12% relative reduction in word error rate (WER) on the LibriSpeech test-other set, improving from 10.1% to 8.8%.