HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Models
This provides a comprehensive solution for 3D hand-object interaction tasks, benefiting applications in robotics and virtual reality, though it is incremental as it builds on existing language model and tokenization techniques.
The paper tackles the problem of unifying 3D hand-object interaction perception and generation by introducing HOIGPT, a method that uses a large language model to generate high-quality sequences from text or partial inputs, achieving state-of-the-art results with improvements like +2.01% R Precision and -2.56 FID.
We introduce HOIGPT, a token-based generative method that unifies 3D hand-object interactions (HOI) perception and generation, offering the first comprehensive solution for captioning and generating high-quality 3D HOI sequences from a diverse range of conditional signals (\eg text, objects, partial sequences). At its core, HOIGPT utilizes a large language model to predict the bidrectional transformation between HOI sequences and natural language descriptions. Given text inputs, HOIGPT generates a sequence of hand and object meshes; given (partial) HOI sequences, HOIGPT generates text descriptions and completes the sequences. To facilitate HOI understanding with a large language model, this paper introduces two key innovations: (1) a novel physically grounded HOI tokenizer, the hand-object decomposed VQ-VAE, for discretizing HOI sequences, and (2) a motion-aware language model trained to process and generate both text and HOI tokens. Extensive experiments demonstrate that HOIGPT sets new state-of-the-art performance on both text generation (+2.01% R Precision) and HOI generation (-2.56 FID) across multiple tasks and benchmarks.