Simpson's Paradox and the Accuracy-Fluency Tradeoff in Translation
This clarifies a foundational issue in machine translation quality assessment, impacting both evaluation methods and system development.
The paper resolves the theoretical puzzle of whether translation accuracy and fluency trade off or correlate by showing it's an instance of Simpson's paradox, with positive correlation at the corpus level but a trade-off at the individual segment level.
A good translation should be faithful to the source and should respect the norms of the target language. We address a theoretical puzzle about the relationship between these objectives. On one hand, intuition and some prior work suggest that accuracy and fluency should trade off against each other, and that capturing every detail of the source can only be achieved at the cost of fluency. On the other hand, quality assessment researchers often suggest that accuracy and fluency are highly correlated and difficult for human raters to distinguish (Callison-Burch et al., 2007). We show that the tension between these views is an instance of Simpson's paradox, and that accuracy and fluency are positively correlated at the level of the corpus but trade off at the level of individual source segments. We further suggest that the relationship between accuracy and fluency is best evaluated at the segment (or sentence) level, and that the trade off between these dimensions has implications both for assessing translation quality and developing improved MT systems.