Decoupled Object-Centric Video Understanding for Generating Robotic Manipulation Commands
For robotic manipulation from video demonstrations, this work addresses the problem of ambiguous commands by explicitly identifying task-relevant objects, achieving large improvements over task-specific baselines.
The paper proposes a decoupled object-centric video understanding framework that separates action recognition from object identification to generate precise manipulation commands. On a modified Something-Something V2 dataset, it achieves 86.79% action accuracy and BLEU-4 scores of 0.337 (standard) and 0.261 (novel), outperforming task-specific baselines by 80-144%.
Translating video demonstrations into executable robot commands remains challenging because existing methods often fail to identify which objects are functionally involved in the demonstrated action. As a result, they may generate commands that are linguistically plausible but operationally ambiguous. We propose an object-centric video understanding framework that decouples action recognition from object identification to generate precise, grammar-free manipulation commands. Our approach integrates Temporal Shift Modules (TSM) for efficient spatio-temporal action classification with a novel \textbf{Object Selection} algorithm that identifies task-relevant objects through trajectory-based role classification, blur detection, and overlap minimization. The selected objects are then processed by Vision-Language Models (VLMs) for robust category recognition and zero-shot generalization. Evaluated on a modified Something-Something V2 dataset, our method achieves 86.79\% action classification accuracy and BLEU-4 scores of 0.337 on standard objects and 0.261 on novel objects. These results improve over the strongest task-specific baseline by 80.2\% and 143.9\%, respectively. Larger gains are observed in METEOR and CIDEr, reaching 157.9\% and 171.7\% on novel objects. Across all semantic metrics, our approach consistently outperforms task-specific methods and remains competitive with, or surpasses, large general-purpose VLMs while retaining a modular, object-centric design.