S-JEPA : Soft Clustering Anchors for Self-Supervised Speech Representation Learning
For self-supervised speech representation learning, S-JEPA provides a more efficient and ambiguity-preserving alternative to hard clustering methods, establishing a new Pareto frontier without offline re-clustering.
S-JEPA introduces a self-supervised speech encoder trained with soft clustering targets from a Gaussian Mixture Model, avoiding offline re-clustering and preserving acoustic ambiguity. It achieves the lowest WER among SSL methods under 90M parameters on SUPERB and matches HuBERT-Base on emotion recognition with half the parameters.
Self-supervised speech encoders are predominantly trained by predicting discrete hard cluster IDs at masked positions, a recipe that collapses acoustic ambiguity at category boundaries and requires interrupting training to re-cluster the entire corpus between iterations. We introduce S-JEPA, a JEPA-style encoder-predictor pair trained to match the soft posteriors of a Gaussian Mixture Model at masked positions via KL divergence. Training runs as one continuous optimization trajectory in two phases: a fixed GMM over MFCC features, then an online GMM over encoder features, with the input layer selected adaptively from a label-free signal, removing both the offline re-cluster step and the hand-tuned choice of which transformer layer to cluster on. Under the SUPERB protocol, S-JEPA achieves the lowest WER among evaluated SSL methods below 90M parameters and matches HuBERT-Base on emotion recognition at roughly half its parameter count, establishing a new Pareto frontier without offline re-clustering or teacher distillation. An analysis of the predictor's per-frame entropy on held-out speech reveals a bimodal distribution with a substantial minority of frames near the entropy of a perfect two-cluster tie, providing direct empirical evidence that the soft-target objective preserves the acoustic ambiguity that hard targets would collapse. Code is available at https://github.com/gioannides/s-jepa.