Skeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models
This addresses the problem of integrating skeleton data into multi-modal action recognition for researchers and practitioners, offering a novel approach but with incremental impact as it adapts existing vision models to a new data type.
The paper tackles the challenge of applying vision-pretrained models to 3D human skeleton data by introducing Skeleton-to-Image Encoding (S2I), which transforms skeleton sequences into image-like formats, enabling self-supervised learning and achieving effective results on datasets like NTU-60, NTU-120, and PKU-MMD.
Recent advances in large-scale pretrained vision models have demonstrated impressive capabilities across a wide range of downstream tasks, including cross-modal and multi-modal scenarios. However, their direct application to 3D human skeleton data remains challenging due to fundamental differences in data format. Moreover, the scarcity of large-scale skeleton datasets and the need to incorporate skeleton data into multi-modal action recognition without introducing additional model branches present significant research opportunities. To address these challenges, we introduce Skeleton-to-Image Encoding (S2I), a novel representation that transforms skeleton sequences into image-like data by partitioning and arranging joints based on body-part semantics and resizing to standardized image dimensions. This encoding enables, for the first time, the use of powerful vision-pretrained models for self-supervised skeleton representation learning, effectively transferring rich visual-domain knowledge to skeleton analysis. While existing skeleton methods often design models tailored to specific, homogeneous skeleton formats, they overlook the structural heterogeneity that naturally arises from diverse data sources. In contrast, our S2I representation offers a unified image-like format that naturally accommodates heterogeneous skeleton data. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate the effectiveness and generalizability of our method for self-supervised skeleton representation learning, including under challenging cross-format evaluation settings.