Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap
This work addresses the need for less intrusive AI interactions in domains like healthcare and education, though it is incremental as it builds on prior LLM-powered agents and HCI studies.
The paper tackles the problem of designing AI assistants that operate unobtrusively by monitoring ambient activity and intervening contextually, establishing the first analysis and taxonomy of 'overhearing LLM agents' as a distinct paradigm in human-AI interaction.
Imagine AI assistants that enhance conversations without interrupting them: quietly providing relevant information during a medical consultation, seamlessly preparing materials as teachers discuss lesson plans, or unobtrusively scheduling meetings as colleagues debate calendars. While modern conversational LLM agents directly assist human users with tasks through a chat interface, we study this alternative paradigm for interacting with LLM agents, which we call "overhearing agents." Rather than demanding the user's attention, overhearing agents continuously monitor ambient activity and intervene only when they can provide contextual assistance. In this paper, we present the first analysis of overhearing LLM agents as a distinct paradigm in human-AI interaction and establish a taxonomy of overhearing agent interactions and tasks grounded in a survey of works on prior LLM-powered agents and exploratory HCI studies. Based on this taxonomy, we create a list of best practices for researchers and developers building overhearing agent systems. Finally, we outline the remaining research gaps and reveal opportunities for future research in the overhearing paradigm.