SUGAR: Learning Skeleton Representation with Visual-Motion Knowledge for Action Recognition
This work addresses the challenge of integrating LLMs with skeleton data for action classification and description, offering a novel approach that could enhance multimodal AI applications, though it appears incremental in combining existing techniques.
The paper tackles the problem of enabling Large Language Models (LLMs) to understand human skeleton data for action recognition by introducing SUGAR, a method that uses visual-motion knowledge from video models to supervise skeleton learning and generate discrete representations for LLMs, achieving efficacy on benchmarks and versatility in zero-shot scenarios.
Large Language Models (LLMs) hold rich implicit knowledge and powerful transferability. In this paper, we explore the combination of LLMs with the human skeleton to perform action classification and description. However, when treating LLM as a recognizer, two questions arise: 1) How can LLMs understand skeleton? 2) How can LLMs distinguish among actions? To address these problems, we introduce a novel paradigm named learning Skeleton representation with visUal-motion knowledGe for Action Recognition (SUGAR). In our pipeline, we first utilize off-the-shelf large-scale video models as a knowledge base to generate visual, motion information related to actions. Then, we propose to supervise skeleton learning through this prior knowledge to yield discrete representations. Finally, we use the LLM with untouched pre-training weights to understand these representations and generate the desired action targets and descriptions. Notably, we present a Temporal Query Projection (TQP) module to continuously model the skeleton signals with long sequences. Experiments on several skeleton-based action classification benchmarks demonstrate the efficacy of our SUGAR. Moreover, experiments on zero-shot scenarios show that SUGAR is more versatile than linear-based methods.