Estimating Subjective Crowd-Evaluations as an Additional Objective to Improve Natural Language Generation
This addresses the challenge of aligning language models with human preferences for natural language generation, though it is incremental as it builds on existing multi-task learning approaches.
The paper tackled the problem of improving natural language generation by incorporating subjective human evaluations as an additional training objective in a multi-task learning setting, resulting in models that generated dialogue lines rated as most typical, most moving the conversation forward, and least offensive in a human evaluation.
Human ratings are one of the most prevalent methods to evaluate the performance of natural language processing algorithms. Similarly, it is common to measure the quality of sentences generated by a natural language generation model using human raters. In this paper, we argue for exploring the use of subjective evaluations within the process of training language generation models in a multi-task learning setting. As a case study, we use a crowd-authored dialogue corpus to fine-tune six different language generation models. Two of these models incorporate multi-task learning and use subjective ratings of lines as part of an explicit learning goal. A human evaluation of the generated dialogue lines reveals that utterances generated by the multi-tasking models were subjectively rated as the most typical, most moving the conversation forward, and least offensive. Based on these promising first results, we discuss future research directions for incorporating subjective human evaluations into language model training and to hence keep the human user in the loop during the development process.