Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning
This addresses the challenge of efficiently using human feedback for reinforcement learning in NLP, though it is incremental as it builds on existing methods.
The study tackled the problem of reinforcement learning from human bandit feedback in sequence-to-sequence tasks like neural machine translation, finding that cardinal feedback is most reliable and learnable, leading to improvements of over 1 BLEU score with small datasets.
We present a study on reinforcement learning (RL) from human bandit feedback for sequence-to-sequence learning, exemplified by the task of bandit neural machine translation (NMT). We investigate the reliability of human bandit feedback, and analyze the influence of reliability on the learnability of a reward estimator, and the effect of the quality of reward estimates on the overall RL task. Our analysis of cardinal (5-point ratings) and ordinal (pairwise preferences) feedback shows that their intra- and inter-annotator $α$-agreement is comparable. Best reliability is obtained for standardized cardinal feedback, and cardinal feedback is also easiest to learn and generalize from. Finally, improvements of over 1 BLEU can be obtained by integrating a regression-based reward estimator trained on cardinal feedback for 800 translations into RL for NMT. This shows that RL is possible even from small amounts of fairly reliable human feedback, pointing to a great potential for applications at larger scale.