Textual Similarity as a Key Metric in Machine Translation Quality Estimation
This addresses the need for more accurate quality estimation in machine translation, though it is incremental as it builds on existing methods by integrating a new metric.
This study tackled the problem of assessing translation reliability in Machine Translation Quality Estimation without reference texts by introducing textual similarity as a new metric, finding it exhibits stronger correlations with human scores than traditional metrics, with specific improvements across multiple language pairs.
Machine Translation (MT) Quality Estimation (QE) assesses translation reliability without reference texts. This study introduces "textual similarity" as a new metric for QE, using sentence transformers and cosine similarity to measure semantic closeness. Analyzing data from the MLQE-PE dataset, we found that textual similarity exhibits stronger correlations with human scores than traditional metrics (hter, model evaluation, sentence probability etc.). Employing GAMMs as a statistical tool, we demonstrated that textual similarity consistently outperforms other metrics across multiple language pairs in predicting human scores. We also found that "hter" actually failed to predict human scores in QE. Our findings highlight the effectiveness of textual similarity as a robust QE metric, recommending its integration with other metrics into QE frameworks and MT system training for improved accuracy and usability.