ChatShop: Interactive Information Seeking with Language Agents
This work addresses the need for more realistic evaluation of interactive information seeking in language agents, though it is incremental as it builds on existing task frameworks.
The paper tackled the problem of evaluating language agents' strategic information seeking by redesigning a web shopping task to include ambiguity and multi-turn interactions, showing that the new task effectively assesses agents' ability to accumulate information through conversations and that simulated shoppers mimic human error patterns.
The desire and ability to seek new information strategically are fundamental to human learning but often overlooked in current language agent evaluation. We analyze a popular web shopping task designed to test language agents' ability to perform strategic exploration and discover that it can be reformulated and solved as a single-turn retrieval task without the need for interactive information seeking. This finding encourages us to rethink realistic constraints on information access that would necessitate strategic information seeking. We then redesign the task to introduce a notion of task ambiguity and the role of a shopper, serving as a dynamic party with whom the agent strategically interacts in an open-ended conversation to make informed decisions. Our experiments demonstrate that the proposed task can effectively evaluate the agent's ability to explore and gradually accumulate information through multi-turn interactions. Additionally, we show that large language model-simulated shoppers serve as a good proxy for real human shoppers, revealing similar error patterns in agents.