Exploring Fluent Query Reformulations with Text-to-Text Transformers and Reinforcement Learning
This work addresses the problem of generating fluent and effective query reformulations for client-facing pipelines and downstream NLP tasks like question answering, benefiting systems that interact with users.
The paper explores query reformulation, aiming to convert noisy text into natural language questions to improve downstream tasks. It introduces QRT5, a text-to-text transformer, and applies policy-based reinforcement learning to it, achieving better sample efficiency and improved readability compared to the previous AQA framework.
Query reformulation aims to alter noisy or ambiguous text sequences into coherent ones closer to natural language questions. This is to prevent errors from propagating in a client-facing pipeline and promote better communication with users. Besides, it is crucial to maintain performance in downstream environments like question answering when rephrased queries are given as input. We show that under the previous framework (AQA), attempts to alter RL algorithms do not bring significant benefits to either reward acquisition or sequence fluency. Instead, we leverage a query-reformulating text-to-text transformer (QRT5) and apply policy-based RL algorithms to further nudge this reformulator and obtain better answers downstream by generating reward-acquiring query trajectories. QRT5 shows better sample efficiency in RL to achieve the same level of QA performance as the previous approach. It can generate reformulations with more readability based on query well-formedness evaluations and can generalize to out-of-sample data. Our framework is demonstrated to be flexible, allowing reward signals to be sourced from different downstream environments such as intent classification.