Evaluation of Habitat Robotics using Large Language Models
This work addresses the problem of selecting effective AI models for embodied robotic tasks, providing comparative benchmarks for researchers in robotics and AI.
The paper evaluated Large Language Models on embodied robotic tasks using the Meta PARTNER benchmark, finding that reasoning models like OpenAI o3-mini outperformed non-reasoning models such as GPT-4o and Llama 3 across various configurations.
This paper focuses on evaluating the effectiveness of Large Language Models at solving embodied robotic tasks using the Meta PARTNER benchmark. Meta PARTNR provides simplified environments and robotic interactions within randomized indoor kitchen scenes. Each randomized kitchen scene is given a task where two robotic agents cooperatively work together to solve the task. We evaluated multiple frontier models on Meta PARTNER environments. Our results indicate that reasoning models like OpenAI o3-mini outperform non-reasoning models like OpenAI GPT-4o and Llama 3 when operating in PARTNR's robotic embodied environments. o3-mini displayed outperform across centralized, decentralized, full observability, and partial observability configurations. This provides a promising avenue of research for embodied robotic development.