Generating Difficult-to-Translate Texts
This addresses the need for better benchmarks to evaluate and improve machine translation models, though it is incremental as it builds on existing iterative probing methods.
The paper tackles the problem of quickly obsolete machine translation benchmarks by proposing MT-breaker, a method that uses a large language model to iteratively refine source texts to increase translation difficulty, resulting in examples that are more challenging for target models while preserving naturalness and diversity, with difficulty transferring to other models and languages.
Machine translation benchmarks sourced from the real world are quickly obsoleted, due to most examples being easy for state-of-the-art translation models. This limits the benchmark's ability to distinguish which model is better or to reveal models' weaknesses. Current methods for creating difficult test cases, such as subsampling or from-scratch synthesis, either fall short of identifying difficult examples or suffer from a lack of diversity and naturalness. Inspired by the iterative process of human experts probing for model failures, we propose MT-breaker, a method where a large language model iteratively refines a source text to increase its translation difficulty. The LLM iteratively queries a target machine translation model to guide its generation of difficult examples. Our approach generates examples that are more challenging for the target MT model while preserving the diversity of natural texts. While the examples are tailored to a particular machine translation model during the generation, the difficulty also transfers to other models and languages.