Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Features
This provides a simple method for controlling voice characteristics in synthesis applications, but it is incremental as it builds on prior work on SSL feature analysis.
The paper tackled the problem of understanding how self-supervised speech models encode speaker characteristics within individual feature dimensions, finding that principal dimensions correlate with attributes like pitch, gender, intensity, and noise, and demonstrating in synthesis experiments that these characteristics can be controlled by modifying corresponding dimensions.
How do speech models trained through self-supervised learning structure their representations? Previous studies have looked at how information is encoded in feature vectors across different layers. But few studies have considered whether speech characteristics are captured within individual dimensions of SSL features. In this paper we specifically look at speaker information using PCA on utterance-averaged representations. Using WavLM, we find that the principal dimension that explains most variance encodes pitch and associated characteristics like gender. Other individual principal dimensions correlate with intensity, noise levels, the second formant, and higher frequency characteristics. Finally, in synthesis experiments we show that most characteristics can be controlled by changing the corresponding dimensions. This provides a simple method to control characteristics of the output voice in synthesis applications.