AttX: Attentive Cross-Connections for Fusion of Wearable Signals in Emotion Recognition
This work addresses emotion recognition using wearable signals, offering an incremental improvement in multimodal fusion techniques for this domain.
The authors tackled the problem of multimodal representation learning from wearable data for emotion recognition by proposing cross-modal attentive connections, a dynamic technique that integrates into any pipeline stage to share information between modalities, resulting in superior or competitive performance compared to state-of-the-art methods on three public datasets.
We propose cross-modal attentive connections, a new dynamic and effective technique for multimodal representation learning from wearable data. Our solution can be integrated into any stage of the pipeline, i.e., after any convolutional layer or block, to create intermediate connections between individual streams responsible for processing each modality. Additionally, our method benefits from two properties. First, it can share information uni-directionally (from one modality to the other) or bi-directionally. Second, it can be integrated into multiple stages at the same time to further allow network gradients to be exchanged in several touch-points. We perform extensive experiments on three public multimodal wearable datasets, WESAD, SWELL-KW, and CASE, and demonstrate that our method can effectively regulate and share information between different modalities to learn better representations. Our experiments further demonstrate that once integrated into simple CNN-based multimodal solutions (2, 3, or 4 modalities), our method can result in superior or competitive performance to state-of-the-art and outperform a variety of baseline uni-modal and classical multimodal methods.