Reconstructing and grounding narrated instructional videos in 3D
This addresses the challenge of 3D understanding and language grounding in instructional videos for applications like augmented reality or robotics, but it is incremental as it builds on existing 3D reconstruction and grounding methods.
The paper tackles the problem of reconstructing objects and grounding narrations in 3D from narrated instructional videos, which involve appearance and language variations, and demonstrates successful reconstruction of car engines and text-to-3D association without manual supervision.
Narrated instructional videos often show and describe manipulations of similar objects, e.g., repairing a particular model of a car or laptop. In this work we aim to reconstruct such objects and to localize associated narrations in 3D. Contrary to the standard scenario of instance-level 3D reconstruction, where identical objects or scenes are present in all views, objects in different instructional videos may have large appearance variations given varying conditions and versions of the same product. Narrations may also have large variation in natural language expressions. We address these challenges by three contributions. First, we propose an approach for correspondence estimation combining learnt local features and dense flow. Second, we design a two-step divide and conquer reconstruction approach where the initial 3D reconstructions of individual videos are combined into a 3D alignment graph. Finally, we propose an unsupervised approach to ground natural language in obtained 3D reconstructions. We demonstrate the effectiveness of our approach for the domain of car maintenance. Given raw instructional videos and no manual supervision, our method successfully reconstructs engines of different car models and associates textual descriptions with corresponding objects in 3D.