EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks
This work addresses the challenge of improving embodied intelligence for daily household tasks, but it is incremental as it builds on existing foundation models with enhanced data collection and calibration methods.
The paper tackled the problem of noisy VLM-based auto-labeling in long-horizon household tasks due to lack of accurate human action labels and spatial annotations, resulting in hallucinations and errors; by introducing EgoTL, a think-aloud capture pipeline with calibrated spatial estimators, it improved long-horizon planning and reasoning, achieving gains in step-wise reasoning, instruction following, and spatial grounding after finetuning foundation models.
Large foundation models have made significant advances in embodied intelligence, enabling synthesis and reasoning over egocentric input for household tasks. However, VLM-based auto-labeling is often noisy because the primary data sources lack accurate human action labels, chain-of-thought (CoT), and spatial annotations; these errors are amplified during long-horizon spatial instruction following. These issues stem from insufficient coverage of minute-long, daily household planning tasks and from inaccurate spatial grounding. As a result, VLM reasoning chains and world-model synthesis can hallucinate objects, skip steps, or fail to respect real-world physical attributes. To address these gaps, we introduce EgoTL. EgoTL builds a think-aloud capture pipeline for egocentric data. It uses a say-before-act protocol to record step-by-step goals and spoken reasoning with word-level timestamps, then calibrates physical properties with metric-scale spatial estimators, a memory-bank walkthrough for scene context, and clip-level tags for navigation instructions and detailed manipulation actions. With EgoTL, we are able to benchmark VLMs and World Models on six task dimensions from three layers and long-horizon generation over minute-long sequences across over 100 daily household tasks. We find that foundation models still fall short as egocentric assistants or open-world simulators. Finally, we finetune foundation models with human CoT aligned with metric labels on the training split of EgoTL, which improves long-horizon planning and reasoning, step-wise reasoning, instruction following, and spatial grounding.