Weakly-supervised Multi-task Learning for Multimodal Affect Recognition
This addresses the challenge of building robust multimodal affect recognition systems for human-computer interaction, though it is incremental as it applies existing multi-task learning techniques to a specific domain.
The paper tackles the problem of limited annotated data for multimodal affect recognition by using weakly-supervised multi-task learning across multiple small datasets, resulting in improvements of up to 2.9% accuracy and 3.3% F1-score for tasks like emotion recognition, sentiment analysis, and sarcasm recognition.
Multimodal affect recognition constitutes an important aspect for enhancing interpersonal relationships in human-computer interaction. However, relevant data is hard to come by and notably costly to annotate, which poses a challenging barrier to build robust multimodal affect recognition systems. Models trained on these relatively small datasets tend to overfit and the improvement gained by using complex state-of-the-art models is marginal compared to simple baselines. Meanwhile, there are many different multimodal affect recognition datasets, though each may be small. In this paper, we propose to leverage these datasets using weakly-supervised multi-task learning to improve the generalization performance on each of them. Specifically, we explore three multimodal affect recognition tasks: 1) emotion recognition; 2) sentiment analysis; and 3) sarcasm recognition. Our experimental results show that multi-tasking can benefit all these tasks, achieving an improvement up to 2.9% accuracy and 3.3% F1-score. Furthermore, our method also helps to improve the stability of model performance. In addition, our analysis suggests that weak supervision can provide a comparable contribution to strong supervision if the tasks are highly correlated.