mmJoints: Expanding Joint Representations Beyond (x,y,z) in mmWave-Based 3D Pose Estimation
This work addresses performance degradation in gesture and activity recognition for mmWave-based pose estimation, offering an incremental improvement by making bias explicit.
The paper tackles the problem of mmWave-based 3D pose estimation, where sparse signals cause models to rely on priors, degrading downstream tasks like activity recognition, and introduces mmJoints to augment pose outputs with joint descriptors, improving joint position accuracy by up to 12.5% and activity recognition by up to 16%.
In mmWave-based pose estimation, sparse signals and weak reflections often cause models to infer body joints from statistical priors rather than sensor data. While prior knowledge helps in learning meaningful representations, over-reliance on it degrades performance in downstream tasks like gesture and activity recognition. In this paper, we introduce mmJoints, a framework that augments a pre-trained, black-box mmWave-based 3D pose estimator's output with additional joint descriptors. Rather than mitigating bias, mmJoints makes it explicit by estimating the likelihood of a joint being sensed and the reliability of its predicted location. These descriptors enhance interpretability and improve downstream task accuracy. Through extensive evaluations using over 115,000 signal frames across 13 pose estimation settings, we show that mmJoints estimates descriptors with an error rate below 4.2%. mmJoints also improves joint position accuracy by up to 12.5% and boosts activity recognition by up to 16% over state-of-the-art methods.