Enabling Rapid Shared Human-AI Mental Model Alignment via the After-Action Review
This work addresses the need for faster and more accessible testing and analysis tools for researchers in human-AI collaboration, though it is incremental as it builds on existing platforms like Minecraft and GPT-4.
The paper tackles the problem of accelerating research in human-machine teaming by introducing a Minecraft testbed for rapid testing of collaborative AI agents with real humans and a tool for post-mission analysis to align mental models, resulting in a browser-based environment that reduces setup time and leverages GPT-4 for querying AI behaviors.
In this work, we present two novel contributions toward improving research in human-machine teaming (HMT): 1) a Minecraft testbed to accelerate testing and deployment of collaborative AI agents and 2) a tool to allow users to revisit and analyze behaviors within an HMT episode to facilitate shared mental model development. Our browser-based Minecraft testbed allows for rapid testing of collaborative agents in a continuous-space, real-time, partially-observable environment with real humans without cumbersome setup typical to human-AI interaction user studies. As Minecraft has an extensive player base and a rich ecosystem of pre-built AI agents, we hope this contribution can help to facilitate research quickly in the design of new collaborative agents and in understanding different human factors within HMT. Our mental model alignment tool facilitates user-led post-mission analysis by including video displays of first-person perspectives of the team members (i.e., the human and AI) that can be replayed, and a chat interface that leverages GPT-4 to provide answers to various queries regarding the AI's experiences and model details.