CAARMA: Class Augmentation with Adversarial Mixup Regularization
This work addresses a domain-specific problem for speaker verification and other zero-shot speech analysis tasks, offering an incremental improvement through synthetic class generation.
The paper tackles the problem of limited class diversity in speaker verification datasets by introducing CAARMA, a class augmentation framework that generates synthetic classes through data mixing in the embedding space, resulting in an 8% improvement over baseline models across multiple tasks.
Speaker verification is a typical zero-shot learning task, where inference of unseen classes is performed by comparing embeddings of test instances to known examples. The models performing inference must hence naturally generate embeddings that cluster same-class instances compactly, while maintaining separation across classes. In order to learn to do so, they are typically trained on a large number of classes (speakers), often using specialized losses. However real-world speaker datasets often lack the class diversity needed to effectively learn this in a generalizable manner. We introduce CAARMA, a class augmentation framework that addresses this problem by generating synthetic classes through data mixing in the embedding space, expanding the number of training classes. To ensure the authenticity of the synthetic classes we adopt a novel adversarial refinement mechanism that minimizes categorical distinctions between synthetic and real classes. We evaluate CAARMA on multiple speaker verification tasks, as well as other representative zero-shot comparison-based speech analysis tasks and obtain consistent improvements: our framework demonstrates a significant improvement of 8\% over all baseline models. The code is available at: https://github.com/massabaali7/CAARMA/