Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations
This addresses the problem of serendipitous learning for users in information-seeking scenarios, representing an incremental improvement over existing methods.
The paper tackles the challenge of discovering unknown unknowns in information-seeking by introducing Co-STORM, a system that lets users observe and steer conversations among language model agents, which outperforms baselines on discourse trace and report quality, with 70% of participants preferring it over a search engine and 78% over a RAG chatbot.
While language model (LM)-powered chatbots and generative search engines excel at answering concrete queries, discovering information in the terrain of unknown unknowns remains challenging for users. To emulate the common educational scenario where children/students learn by listening to and participating in conversations of their parents/teachers, we create Collaborative STORM (Co-STORM). Unlike QA systems that require users to ask all the questions, Co-STORM lets users observe and occasionally steer the discourse among several LM agents. The agents ask questions on the user's behalf, allowing the user to discover unknown unknowns serendipitously. To facilitate user interaction, Co-STORM assists users in tracking the discourse by organizing the uncovered information into a dynamic mind map, ultimately generating a comprehensive report as takeaways. For automatic evaluation, we construct the WildSeek dataset by collecting real information-seeking records with user goals. Co-STORM outperforms baseline methods on both discourse trace and report quality. In a further human evaluation, 70% of participants prefer Co-STORM over a search engine, and 78% favor it over a RAG chatbot.