Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments
This addresses the problem of instruction following for AI agents in virtual settings, representing an incremental advancement by integrating existing methods.
The study tackled enabling AI agents to execute complex language instructions in virtual environments by proposing a hierarchical framework combining large language models for comprehension and reinforcement learning for action execution, demonstrating effectiveness in IGLU and Crafter environments with unspecified performance metrics.
In this study, we address the issue of enabling an artificial intelligence agent to execute complex language instructions within virtual environments. In our framework, we assume that these instructions involve intricate linguistic structures and multiple interdependent tasks that must be navigated successfully to achieve the desired outcomes. To effectively manage these complexities, we propose a hierarchical framework that combines the deep language comprehension of large language models with the adaptive action-execution capabilities of reinforcement learning agents. The language module (based on LLM) translates the language instruction into a high-level action plan, which is then executed by a pre-trained reinforcement learning agent. We have demonstrated the effectiveness of our approach in two different environments: in IGLU, where agents are instructed to build structures, and in Crafter, where agents perform tasks and interact with objects in the surrounding environment according to language commands.