Interactive Text Ranking with Bayesian Optimisation: A Case Study on Community QA and Summarisation
This addresses the lack of user-specific data for interactive text ranking in NLP applications like community QA and summarization, offering a novel method for efficient user feedback integration.
The paper tackles the problem of selecting the best text from many candidates in NLP tasks like question answering and summarization without user-specific training data by proposing an interactive ranking method using Bayesian optimization to focus user labeling on high-quality candidates. It significantly outperforms existing interactive approaches in community QA and summarization, and the learned ranking function improves state-of-the-art in interactive summarization as a reward function for reinforcement learning.
For many NLP applications, such as question answering and summarisation, the goal is to select the best solution from a large space of candidates to meet a particular user's needs. To address the lack of user-specific training data, we propose an interactive text ranking approach that actively selects pairs of candidates, from which the user selects the best. Unlike previous strategies, which attempt to learn a ranking across the whole candidate space, our method employs Bayesian optimisation to focus the user's labelling effort on high quality candidates and integrates prior knowledge in a Bayesian manner to cope better with small data scenarios. We apply our method to community question answering (cQA) and extractive summarisation, finding that it significantly outperforms existing interactive approaches. We also show that the ranking function learned by our method is an effective reward function for reinforcement learning, which improves the state of the art for interactive summarisation.