A Real-Time System for Egocentric Hand-Object Interaction Detection in Industrial Domains
This addresses the need for fast and accurate interaction detection in industrial domains, but it is incremental as it combines existing methods in a cascaded architecture.
The paper tackles real-time detection of hand-object interactions in industrial settings using egocentric vision, achieving 38.52% p-AP on action recognition and 85.13% AP for object detection at 30fps.
Hand-object interaction detection remains an open challenge in real-time applications, where intuitive user experiences depend on fast and accurate detection of interactions with surrounding objects. We propose an efficient approach for detecting hand-objects interactions from streaming egocentric vision that operates in real time. Our approach consists of an action recognition module and an object detection module for identifying active objects upon confirmed interaction. Our Mamba model with EfficientNetV2 as backbone for action recognition achieves 38.52% p-AP on the ENIGMA-51 benchmark at 30fps, while our fine-tuned YOLOWorld reaches 85.13% AP for hand and object. We implement our models in a cascaded architecture where the action recognition and object detection modules operate sequentially. When the action recognition predicts a contact state, it activates the object detection module, which in turn performs inference on the relevant frame to detect and classify the active object.