What do the metrics mean? A critical analysis of the use of Automated Evaluation Metrics in Interpreting
This addresses the problem of efficiently measuring interpreting quality for researchers and practitioners, but it is incremental as it critiques existing methods without proposing a new solution.
The paper critically analyzes the use of automated evaluation metrics in interpreting, concluding that current metrics fail to account for communicative context and are not viable standalone measures of interpreting quality.
With the growth of interpreting technologies, from remote interpreting and Computer-Aided Interpreting to automated speech translation and interpreting avatars, there is now a high demand for ways to quickly and efficiently measure the quality of any interpreting delivered. A range of approaches to fulfil the need for quick and efficient quality measurement have been proposed, each involving some measure of automation. This article examines these recently-proposed quality measurement methods and will discuss their suitability for measuring the quality of authentic interpreting practice, whether delivered by humans or machines, concluding that automatic metrics as currently proposed cannot take into account the communicative context and thus are not viable measures of the quality of any interpreting provision when used on their own. Across all attempts to measure or even categorise quality in Interpreting Studies, the contexts in which interpreting takes place have become fundamental to the final analysis.