Tachikuma: Understading Complex Interactions with Multi-Character and Novel Objects by Large Language Models
This work addresses the problem of enhancing AI agents' ability to simulate complex human-like interactions in virtual environments, which is incremental as it builds on existing LLM capabilities by adding structured oversight mechanisms.
The paper tackles the limitations of AI agents in handling complex interactions involving multiple characters and novel objects by proposing to integrate virtual Game Masters into the agent's world model, inspired by Tabletop Role-Playing Games, and introduces a benchmark called Tachikuma with a dataset and task to evaluate and improve interaction understanding.
Recent advancements in natural language and Large Language Models (LLMs) have enabled AI agents to simulate human-like interactions within virtual worlds. However, these interactions still face limitations in complexity and flexibility, particularly in scenarios involving multiple characters and novel objects. Pre-defining all interactable objects in the agent's world model presents challenges, and conveying implicit intentions to multiple characters through complex interactions remains difficult. To address these issues, we propose integrating virtual Game Masters (GMs) into the agent's world model, drawing inspiration from Tabletop Role-Playing Games (TRPGs). GMs play a crucial role in overseeing information, estimating players' intentions, providing environment descriptions, and offering feedback, compensating for current world model deficiencies. To facilitate future explorations for complex interactions, we introduce a benchmark named Tachikuma, comprising a Multiple character and novel Object based interaction Estimation (MOE) task and a supporting dataset. MOE challenges models to understand characters' intentions and accurately determine their actions within intricate contexts involving multi-character and novel object interactions. Besides, the dataset captures log data from real-time communications during gameplay, providing diverse, grounded, and complex interactions for further explorations. Finally, we present a simple prompting baseline and evaluate its performance, demonstrating its effectiveness in enhancing interaction understanding. We hope that our dataset and task will inspire further research in complex interactions with natural language, fostering the development of more advanced AI agents.