Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
This work addresses the design of AI-mediated communication systems for tasks like handling constituent emails, offering incremental insights into user experience trade-offs.
The study compared sentence-level and message-level AI suggestions for email responses, finding that message-level suggestions led to faster responses and higher satisfaction, while sentence-level suggestions increased user agency but took longer.
Traditionally, writing assistance systems have focused on short or even single-word suggestions. Recently, large language models like GPT-3 have made it possible to generate significantly longer natural-sounding suggestions, offering more advanced assistance opportunities. This study explores the trade-offs between sentence- vs. message-level suggestions for AI-mediated communication. We recruited 120 participants to act as staffers from legislators' offices who often need to respond to large volumes of constituent concerns. Participants were asked to reply to emails with different types of assistance. The results show that participants receiving message-level suggestions responded faster and were more satisfied with the experience, as they mainly edited the suggested drafts. In addition, the texts they wrote were evaluated as more helpful by others. In comparison, participants receiving sentence-level assistance retained a higher sense of agency, but took longer for the task as they needed to plan the flow of their responses and decide when to use suggestions. Our findings have implications for designing task-appropriate communication assistance systems.