Improving and Evaluating Hand-Object Interaction Detection
For researchers in action perception, 3D reconstruction, and robotics, this work provides a significantly improved method for detecting hand-object interactions, though it is an incremental improvement over existing DETR-based approaches.
The paper introduces HOI-DETR, a new framework that integrates hand-object and object-object interactions into the Co-DETR architecture, achieving state-of-the-art results on multiple HOI benchmarks with mAP gains exceeding 20 percentage points on Hands23 and FineBio.
Understanding hands and the objects they interact with, both directly and through tools, is a key step for tasks ranging from action perception to 3D reconstruction and robotics. Our paper provides several contributions to the Hand-Object Interaction (HOI) understanding literature: (1) HOI-DETR, a new framework that introduces hand-object and object-object interactions to the Co-DETR architecture to produce a state-of-the-art method; (2) a comprehensive HOI evaluation suite of 4 diverse datasets, including a video benchmark derived from the HD-EPIC dataset and fresh annotations that improve the Hands23 benchmark and (3) a trained checkpoint that significantly improves the state of the art across Hands23, HOIST, FineBio, and HD-EPIC, including mAP gains of over 20 percentage points on Hands23 and FineBio. Our ablations confirm the contributions of each model component.