Cascaded Cross-Modal Transformer for Request and Complaint Detection
This work addresses a domain-specific problem in customer service analytics, offering incremental improvements in multimodal detection for phone conversations.
The paper tackles the problem of detecting customer requests and complaints in phone conversations by proposing a cascaded cross-modal transformer that combines speech and text transcripts, achieving unweighted average recalls of 65.41% for complaints and 85.87% for requests.
We propose a novel cascaded cross-modal transformer (CCMT) that combines speech and text transcripts to detect customer requests and complaints in phone conversations. Our approach leverages a multimodal paradigm by transcribing the speech using automatic speech recognition (ASR) models and translating the transcripts into different languages. Subsequently, we combine language-specific BERT-based models with Wav2Vec2.0 audio features in a novel cascaded cross-attention transformer model. We apply our system to the Requests Sub-Challenge of the ACM Multimedia 2023 Computational Paralinguistics Challenge, reaching unweighted average recalls (UAR) of 65.41% and 85.87% for the complaint and request classes, respectively.