Automatic Evaluation of Generative Models with Instruction Tuning
This addresses the challenge of evaluating generative models in NLP, offering a method that could reduce reliance on human annotations, but it is incremental as it builds on existing instruction-tuning paradigms.
The paper tackles the problem of automatic evaluation for natural language generation by proposing a learned metric based on instruction tuning, which achieves good performance on various tasks using the HEAP dataset, though some criteria are harder to learn.
Automatic evaluation of natural language generation has long been an elusive goal in NLP.A recent paradigm fine-tunes pre-trained language models to emulate human judgements for a particular task and evaluation criterion. Inspired by the generalization ability of instruction-tuned models, we propose a learned metric based on instruction tuning. To test our approach, we collected HEAP, a dataset of human judgements across various NLG tasks and evaluation criteria. Our findings demonstrate that instruction tuning language models on HEAP yields good performance on many evaluation tasks, though some criteria are less trivial to learn than others. Further, jointly training on multiple tasks can yield additional performance improvements, which can be beneficial for future tasks with little to no human annotated data.