Evaluating MT Systems: A Theoretical Framework
This work addresses the need for better evaluation metrics in machine translation, particularly for emerging applications, but it is incremental as it builds on existing frameworks without introducing a new method.
The paper tackles the problem of evaluating machine translation systems by proposing a theoretical framework centered on 'cognitive ease', which is derived from adequacy and lack of fluency, to design automatic metrics. It aims to improve understanding and future enhancements of existing methods and apply to newer MT types like speech-to-speech translation.
This paper outlines a theoretical framework using which different automatic metrics can be designed for evaluation of Machine Translation systems. It introduces the concept of {\em cognitive ease} which depends on {\em adequacy} and {\em lack of fluency}. Thus, cognitive ease becomes the main parameter to be measured rather than comprehensibility. The framework allows the components of cognitive ease to be broken up and computed based on different linguistic levels etc. Independence of dimensions and linearly combining them provides for a highly modular approach. The paper places the existing automatic methods in an overall framework, to understand them better and to improve upon them in future. It can also be used to evaluate the newer types of MT systems, such as speech to speech translation and discourse translation.