EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates
For embodied AI researchers, this provides a new benchmark and dataset for learning interactive object models from real-world videos, addressing a gap in available simulation-ready objects.
EgoFun3D introduces a task, dataset, and benchmark for modeling interactive 3D objects from egocentric videos, using function templates to capture cross-part functional mappings. The dataset includes 271 videos with annotations, and benchmarking shows the task is challenging for existing methods.
We present EgoFun3D, a coordinated task formulation, dataset, and benchmark for modeling interactive 3D objects from egocentric videos. Interactive objects are of high interest for embodied AI but scarce, making modeling from readily available real-world videos valuable. Our task focuses on obtaining simulation-ready interactive 3D objects from egocentric video input. While prior work largely focuses on articulations, we capture general cross-part functional mappings (e.g., rotation of stove knob controls stove burner temperature) through function templates, a structured computational representation. Function templates enable precise evaluation and direct compilation into executable code across simulation platforms. To enable comprehensive benchmarking, we introduce a dataset of 271 egocentric videos featuring challenging real-world interactions with paired 3D geometry, segmentation over 2D and 3D, articulation and function template annotations. To tackle the task, we propose a 4-stage pipeline consisting of: 2D part segmentation, reconstruction, articulation estimation, and function template inference. Comprehensive benchmarking shows that the task is challenging for off-the-shelf methods, highlighting avenues for future work.