Human Motion Instruction Tuning
This work addresses the need for more accurate human motion analysis in applications like sports analytics and behavioral prediction, though it appears incremental as it builds on existing instruction-tuning approaches.
The paper tackles the problem of preserving motion-specific details in multimodal instruction tuning by introducing LLaMo, a framework that retains motion in its native form instead of converting it to language tokens, resulting in improved interpretation of complex human behaviors in high-complexity domains.
This paper presents LLaMo (Large Language and Human Motion Assistant), a multimodal framework for human motion instruction tuning. In contrast to conventional instruction-tuning approaches that convert non-linguistic inputs, such as video or motion sequences, into language tokens, LLaMo retains motion in its native form for instruction tuning. This method preserves motion-specific details that are often diminished in tokenization, thereby improving the model's ability to interpret complex human behaviors. By processing both video and motion data alongside textual inputs, LLaMo enables a flexible, human-centric analysis. Experimental evaluations across high-complexity domains, including human behaviors and professional activities, indicate that LLaMo effectively captures domain-specific knowledge, enhancing comprehension and prediction in motion-intensive scenarios. We hope LLaMo offers a foundation for future multimodal AI systems with broad applications, from sports analytics to behavioral prediction. Our code and models are available on the project website: https://github.com/ILGLJ/LLaMo.