Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT
This addresses the problem of user trust in imperfect machine translation for deployment scenarios, though it is a pilot study and thus incremental.
The study investigated how users adjust their trust in machine translation systems when encountering errors, finding that users reacted strongly to disfluent translations but were surprisingly less concerned with adequacy issues.
Although measuring intrinsic quality has been a key factor in the advancement of Machine Translation (MT), successfully deploying MT requires considering not just intrinsic quality but also the user experience, including aspects such as trust. This work introduces a method of studying how users modulate their trust in an MT system after seeing errorful (disfluent or inadequate) output amidst good (fluent and adequate) output. We conduct a survey to determine how users respond to good translations compared to translations that are either adequate but not fluent, or fluent but not adequate. In this pilot study, users responded strongly to disfluent translations, but were, surprisingly, much less concerned with adequacy.