Inertial Hallucinations -- When Wearable Inertial Devices Start Seeing Things
This work addresses sensor fusion challenges in Ambient Assisted Living, offering incremental improvements for wearable inertial device applications.
The paper tackled the problem of limited coverage and reliability in multimodal sensor fusion for Ambient Assisted Living by proposing a framework that fuses modality hallucination with triplet learning to handle missing sensors at inference time, resulting in an average accuracy improvement of 6.6% on UTD-MHAD and 5.5% on Berkeley MHAD datasets, achieving new state-of-the-art inertial-only classification accuracy.
We propose a novel approach to multimodal sensor fusion for Ambient Assisted Living (AAL) which takes advantage of learning using privileged information (LUPI). We address two major shortcomings of standard multimodal approaches, limited area coverage and reduced reliability. Our new framework fuses the concept of modality hallucination with triplet learning to train a model with different modalities to handle missing sensors at inference time. We evaluate the proposed model on inertial data from a wearable accelerometer device, using RGB videos and skeletons as privileged modalities, and show an improvement of accuracy of an average 6.6% on the UTD-MHAD dataset and an average 5.5% on the Berkeley MHAD dataset, reaching a new state-of-the-art for inertial-only classification accuracy on these datasets. We validate our framework through several ablation studies.