Follow-Up Questions Improve Documents Generated by Large Language Models
This addresses the problem of unclear user requests in AI-generated documents for users, though it is incremental as it builds on existing LLM capabilities.
This study tackled the problem of improving document generation by Large Language Models (LLMs) by having them ask follow-up questions to clarify user needs, resulting in users preferring documents generated with this interactive approach and valuing thought-provoking questions.
This study investigates the impact of Large Language Models (LLMs) generating follow-up questions in response to user requests for short (1-page) text documents. Users interacted with a novel web-based AI system designed to ask follow-up questions. Users requested documents they would like the AI to produce. The AI then generated follow-up questions to clarify the user's needs or offer additional insights before generating the requested documents. After answering the questions, users were shown a document generated using both the initial request and the questions and answers, and a document generated using only the initial request. Users indicated which document they preferred and gave feedback about their experience with the question-answering process. The findings of this study show clear benefits to question-asking both in document preference and in the qualitative user experience. This study further shows that users found more value in questions which were thought-provoking, open-ended, or offered unique insights into the user's request as opposed to simple information-gathering questions.