Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration
This addresses the need for human-agent collaboration in scenarios requiring human preferences or expertise, though it is incremental as it builds on existing LM agent frameworks.
The authors tackled the problem of enabling and evaluating human-agent collaboration by introducing Collaborative Gym (Co-Gym), a framework for asynchronous tripartite interactions, and found that collaborative agents outperformed fully autonomous ones in tasks like Travel Planning (86% win rate), Tabular Analysis (74%), and Related Work (66%).
Recent advancements in language models (LMs) have sparked growing interest in developing LM agents. While fully autonomous agents could excel in many scenarios, numerous use cases inherently require them to collaborate with humans due to humans' latent preferences, domain expertise, or need for control. To facilitate the study of human-agent collaboration, we present Collaborative Gym (Co-Gym), a general framework enabling asynchronous, tripartite interaction among agents, humans, and task environments. We instantiate Co-Gym with three representative tasks in both simulated and real-world conditions, and propose an evaluation framework that assesses both the collaboration outcomes and processes. Our findings reveal that collaborative agents consistently outperform their fully autonomous counterparts in task performance within those delivered cases, achieving win rates of 86% in Travel Planning, 74% in Tabular Analysis, and 66% in Related Work when evaluated by real users. However, our study also highlights significant challenges in developing collaborative agents, requiring advancements in core aspects of intelligence -- communication capabilities, situational awareness, and balancing autonomy and human control.