See and Think: Embodied Agent in Virtual Environment
This work addresses the challenge of creating effective embodied agents for virtual environments like Minecraft, representing an incremental advancement in applying LLMs to embodied AI tasks.
The paper tackles the problem of building an embodied agent in the Minecraft virtual environment by proposing STEVE, which integrates vision perception, language instruction, and code action, resulting in up to 1.5x faster unlocking of key tech trees and 2.5x quicker block search tasks.
Large language models (LLMs) have achieved impressive pro-gress on several open-world tasks. Recently, using LLMs to build embodied agents has been a hotspot. This paper proposes STEVE, a comprehensive and visionary embodied agent in the Minecraft virtual environment. STEVE comprises three key components: vision perception, language instruction, and code action. Vision perception involves interpreting visual information in the environment, which is then integrated into the LLMs component with agent state and task instruction. Language instruction is responsible for iterative reasoning and decomposing complex tasks into manageable guidelines. Code action generates executable skill actions based on retrieval in skill database, enabling the agent to interact effectively within the Minecraft environment. We also collect STEVE-21K dataset, which includes 600+ vision-environment pairs, 20K knowledge question-answering pairs, and 200+ skill-code pairs. We conduct continuous block search, knowledge question and answering, and tech tree mastery to evaluate the performance. Extensive experiments show that STEVE achieves at most 1.5x faster unlocking key tech trees and 2.5x quicker in block search tasks.