VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Living
This work addresses the challenge of fine-grained ADL recognition for real-world applications requiring low latency, though it is incremental as it builds on existing pose-driven attention mechanisms.
The paper tackles the problem of recognizing Activities of Daily Living (ADL) by combining RGB and 3D poses, proposing VPN++ which integrates feature-level and attention-level distillation to transfer pose knowledge into RGB, resulting in improved performance and high speed-up with resilience to noisy poses, outperforming baselines on 4 public datasets.
Many attempts have been made towards combining RGB and 3D poses for the recognition of Activities of Daily Living (ADL). ADL may look very similar and often necessitate to model fine-grained details to distinguish them. Because the recent 3D ConvNets are too rigid to capture the subtle visual patterns across an action, this research direction is dominated by methods combining RGB and 3D Poses. But the cost of computing 3D poses from RGB stream is high in the absence of appropriate sensors. This limits the usage of aforementioned approaches in real-world applications requiring low latency. Then, how to best take advantage of 3D Poses for recognizing ADL? To this end, we propose an extension of a pose driven attention mechanism: Video-Pose Network (VPN), exploring two distinct directions. One is to transfer the Pose knowledge into RGB through a feature-level distillation and the other towards mimicking pose driven attention through an attention-level distillation. Finally, these two approaches are integrated into a single model, we call VPN++. We show that VPN++ is not only effective but also provides a high speed up and high resilience to noisy Poses. VPN++, with or without 3D Poses, outperforms the representative baselines on 4 public datasets. Code is available at https://github.com/srijandas07/vpnplusplus.