LM Agents for Coordinating Multi-User Information Gathering
This addresses the problem of evaluating LM agents in realistic multi-user collaboration scenarios for researchers, though it is incremental as it adapts existing benchmarks.
The paper introduces PeopleJoin, a benchmark for evaluating language model agents in multi-user collaborative problem solving, where agents must identify teammates, gather distributed information, and compile answers. They implemented several LM agent architectures and evaluated their accuracy and efficiency on tasks adapted from existing NLP benchmarks.
This paper introduces PeopleJoin, a benchmark for evaluating LM-mediated collaborative problem solving. Given a user request, PeopleJoin agents must identify teammates who might be able to assist, converse with these teammates to gather information, and finally compile a useful answer or summary for the original user. PeopleJoin comprises two evaluation domains: PeopleJoin-QA, focused on questions about tabular data, and PeopleJoin-DocCreation, focused on document creation tasks. The two domains are adapted from existing NLP benchmarks for database question answering and multi-document summarization; here, however, the information needed to complete these tasks is distributed across synthetic ``organizations'' of 2--20 users, simulating natural multi-user collaboration scenarios. We implemented several popular LM agent architectures, evaluating their accuracy and efficiency at completing tasks, and highlight new research questions that can be studied using PeopleJoin.