Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition
This work addresses the challenge of multimodal fusion and data scarcity in wearable emotion recognition, which is important for applications in real-life, less invasive monitoring, but it appears incremental as it builds on existing self-supervised and transformer-based methods.
The authors tackled the problem of wearable emotion recognition by proposing a self-supervised learning framework that fuses multimodal physiological signals using temporal convolution and transformer encoders, achieving state-of-the-art results in emotion classification tasks and improved accuracy and robustness in low-data scenarios.
Recently, wearable emotion recognition based on peripheral physiological signals has drawn massive attention due to its less invasive nature and its applicability in real-life scenarios. However, how to effectively fuse multimodal data remains a challenging problem. Moreover, traditional fully-supervised based approaches suffer from overfitting given limited labeled data. To address the above issues, we propose a novel self-supervised learning (SSL) framework for wearable emotion recognition, where efficient multimodal fusion is realized with temporal convolution-based modality-specific encoders and a transformer-based shared encoder, capturing both intra-modal and inter-modal correlations. Extensive unlabeled data is automatically assigned labels by five signal transforms, and the proposed SSL model is pre-trained with signal transformation recognition as a pretext task, allowing the extraction of generalized multimodal representations for emotion-related downstream tasks. For evaluation, the proposed SSL model was first pre-trained on a large-scale self-collected physiological dataset and the resulting encoder was subsequently frozen or fine-tuned on three public supervised emotion recognition datasets. Ultimately, our SSL-based method achieved state-of-the-art results in various emotion classification tasks. Meanwhile, the proposed model proved to be more accurate and robust compared to fully-supervised methods on low data regimes.