ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining
This addresses the need for richer stylistic language-audio pretraining in speech processing, offering improvements over existing models that handle only a narrow set of descriptors, though it appears incremental in extending contrastive learning to more style dimensions.
The authors tackled the problem of mapping speech and text style captions into a common embedding space to handle a wide range of intrinsic and situational descriptors, and found that specialized models yield stronger performance on individual style dimensions while a unified model excels on compositional evaluation, outperforming baselines on most metrics across style caption retrieval, speech attribute classification, and as an inference-time reward model for TTS.
We introduce ParaSpeechCLAP, a dual-encoder contrastive model that maps speech and text style captions into a common embedding space, supporting a wide range of intrinsic (speaker-level) and situational (utterance-level) descriptors (such as pitch, texture and emotion) far beyond the narrow set handled by existing models. We train specialized ParaSpeechCLAP-Intrinsic and ParaSpeechCLAP-Situational models alongside a unified ParaSpeechCLAP-Combined model, finding that specialization yields stronger performance on individual style dimensions while the unified model excels on compositional evaluation. We further show that ParaSpeechCLAP-Intrinsic benefits from an additional classification loss and class-balanced training. We demonstrate our models' performance on style caption retrieval, speech attribute classification and as an inference-time reward model that improves style-prompted TTS without additional training. ParaSpeechCLAP outperforms baselines on most metrics across all three applications. Our models and code are released at https://github.com/ajd12342/paraspeechclap .