Intelligent Hybrid Man-Machine Translation Quality Estimation
This work addresses the scarcity and inconsistency of expert human judgments in machine translation, which is crucial for real-time applications like post-editing, though it appears incremental in nature.
The paper tackles the problem of expensive and inconsistent human judgments for machine translation quality estimation by introducing a hybrid approach that combines probabilistic inference on human ranks with linguistic features. Experimental results show improved correlation with human judgments over traditional metrics on challenging language pairs.
Inferring evaluation scores based on human judgments is invaluable compared to using current evaluation metrics which are not suitable for real-time applications e.g. post-editing. However, these judgments are much more expensive to collect especially from expert translators, compared to evaluation based on indicators contrasting source and translation texts. This work introduces a novel approach for quality estimation by combining learnt confidence scores from a probabilistic inference model based on human judgments, with selective linguistic features-based scores, where the proposed inference model infers the credibility of given human ranks to solve the scarcity and inconsistency issues of human judgments. Experimental results, using challenging language-pairs, demonstrate improvement in correlation with human judgments over traditional evaluation metrics.