MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels
This addresses the challenge of label scarcity and modality limitations in HAR for emerging applications, representing a novel method for a known bottleneck.
The paper tackles the problem of human activity recognition (HAR) with limited labels and modality constraints by proposing MESEN, a framework that uses unlabeled multimodal data during pre-training to enhance unimodal HAR, achieving significant performance improvements over state-of-the-art baselines in experiments on eight datasets.
Human activity recognition (HAR) will be an essential function of various emerging applications. However, HAR typically encounters challenges related to modality limitations and label scarcity, leading to an application gap between current solutions and real-world requirements. In this work, we propose MESEN, a multimodal-empowered unimodal sensing framework, to utilize unlabeled multimodal data available during the HAR model design phase for unimodal HAR enhancement during the deployment phase. From a study on the impact of supervised multimodal fusion on unimodal feature extraction, MESEN is designed to feature a multi-task mechanism during the multimodal-aided pre-training stage. With the proposed mechanism integrating cross-modal feature contrastive learning and multimodal pseudo-classification aligning, MESEN exploits unlabeled multimodal data to extract effective unimodal features for each modality. Subsequently, MESEN can adapt to downstream unimodal HAR with only a few labeled samples. Extensive experiments on eight public multimodal datasets demonstrate that MESEN achieves significant performance improvements over state-of-the-art baselines in enhancing unimodal HAR by exploiting multimodal data.