Putting People in LLMs' Shoes: Generating Better Answers via Question Rewriter
This work addresses a practical issue in LLM-based question answering by providing a method to enhance question clarity, though it is incremental as it builds on existing prompt optimization techniques.
The paper tackles the problem of vague user questions undermining LLM performance in question answering by introducing a question rewriter that optimizes prompts to improve answer quality, achieving significant improvements across multiple LLMs and datasets.
Large Language Models (LLMs) have demonstrated significant capabilities, particularly in the domain of question answering (QA). However, their effectiveness in QA is often undermined by the vagueness of user questions. To address this issue, we introduce single-round instance-level prompt optimization, referred to as question rewriter. By enhancing the intelligibility of human questions for black-box LLMs, our question rewriter improves the quality of generated answers. The rewriter is optimized using direct preference optimization based on feedback collected from automatic criteria for evaluating generated answers; therefore, its training does not require costly human annotations. The experiments across multiple black-box LLMs and long-form question answering (LFQA) datasets demonstrate the efficacy of our method. This paper provides a practical framework for training question rewriters and sets a precedent for future explorations in prompt optimization within LFQA tasks. Code is available at https://github.com/3244we/Question-Rewriter.