OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL
This work addresses the need for versatile speech representations that support both analysis and synthesis tasks, offering a unified framework for self-supervised learning.
OLIVE proposes a self-supervised speech representation learning framework that jointly optimizes masked latent prediction and waveform reconstruction, achieving improved performance on generation and speaker tasks while maintaining competitive results on recognition and semantic tasks.
We propose Online Latent prediction with Invariant Views and rEconstruction (OLIVE), a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE combines view-augmented masked latent prediction with waveform reconstruction under a unified objective. Reconstruction constrains early encoder features to retain signal-level information, while masked latent prediction shapes later contextual representations toward invariance for robust downstream performance. We show that these objectives enable representations that support a broad range of tasks. In particular, OLIVE improves results on generation and speaker tasks, maintains competitive performance on recognition and semantic tasks, and improves waveform reconstruction.