HEVAL: Yet Another Human Evaluation Metric
This work addresses the need for reliable human evaluation in machine translation development, though it appears incremental as it builds on existing human evaluation methods.
The paper tackles the problem of evaluating machine translation quality by proposing a new human evaluation metric that addresses issues of inter-annotator agreement and repeatability, providing a basis for assessing translation quality.
Machine translation evaluation is a very important activity in machine translation development. Automatic evaluation metrics proposed in literature are inadequate as they require one or more human reference translations to compare them with output produced by machine translation. This does not always give accurate results as a text can have several different translations. Human evaluation metrics, on the other hand, lacks inter-annotator agreement and repeatability. In this paper we have proposed a new human evaluation metric which addresses these issues. Moreover this metric also provides solid grounds for making sound assumptions on the quality of the text produced by a machine translation.