Thinker: A vision-language foundation model for embodied intelligence
This addresses challenges in applying vision-language models to robotics, offering improvements for embodied AI systems, though it appears incremental in its approach.
The paper tackles the problem of large vision-language models making errors in robotics tasks, such as perspective confusion and overlooking video endings, by proposing Thinker, a foundation model for embodied intelligence that achieves state-of-the-art results on two benchmark datasets for task planning.
When large vision-language models are applied to the field of robotics, they encounter problems that are simple for humans yet error-prone for models. Such issues include confusion between third-person and first-person perspectives and a tendency to overlook information in video endings during temporal reasoning. To address these challenges, we propose Thinker, a large vision-language foundation model designed for embodied intelligence. We tackle the aforementioned issues from two perspectives. Firstly, we construct a large-scale dataset tailored for robotic perception and reasoning, encompassing ego-view videos, visual grounding, spatial understanding, and chain-of-thought data. Secondly, we introduce a simple yet effective approach that substantially enhances the model's capacity for video comprehension by jointly incorporating key frames and full video sequences as inputs. Our model achieves state-of-the-art results on two of the most commonly used benchmark datasets in the field of task planning.