Multimodal Skeleton-Based Action Representation Learning via Decomposition and Composition
This addresses the problem of computational overhead in multimodal action recognition for computer vision applications, representing an incremental improvement over existing fusion methods.
The paper tackles the challenge of balancing efficiency and effectiveness in multimodal human action understanding by introducing a self-supervised framework that decomposes fused features into unimodal ones and composes them to enhance learning, achieving strong performance on datasets like NTU RGB+D 60, 120, and PKU-MMD II.
Multimodal human action understanding is a significant problem in computer vision, with the central challenge being the effective utilization of the complementarity among diverse modalities while maintaining model efficiency. However, most existing methods rely on simple late fusion to enhance performance, which results in substantial computational overhead. Although early fusion with a shared backbone for all modalities is efficient, it struggles to achieve excellent performance. To address the dilemma of balancing efficiency and effectiveness, we introduce a self-supervised multimodal skeleton-based action representation learning framework, named Decomposition and Composition. The Decomposition strategy meticulously decomposes the fused multimodal features into distinct unimodal features, subsequently aligning them with their respective ground truth unimodal counterparts. On the other hand, the Composition strategy integrates multiple unimodal features, leveraging them as self-supervised guidance to enhance the learning of multimodal representations. Extensive experiments on the NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD II datasets demonstrate that the proposed method strikes an excellent balance between computational cost and model performance.