FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation
This work addresses the need for standardized evaluation in style-targeted translation for specific regional audiences, but it is incremental as it builds on existing translation benchmarks.
The authors introduced FRMT, a dataset and benchmark for few-shot region-aware machine translation, focusing on English to Portuguese and Mandarin Chinese regional variants, and provided baseline models and evaluation guidelines.
We present FRMT, a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation, a type of style-targeted translation. The dataset consists of professional translations from English into two regional variants each of Portuguese and Mandarin Chinese. Source documents are selected to enable detailed analysis of phenomena of interest, including lexically distinct terms and distractor terms. We explore automatic evaluation metrics for FRMT and validate their correlation with expert human evaluation across both region-matched and mismatched rating scenarios. Finally, we present a number of baseline models for this task, and offer guidelines for how researchers can train, evaluate, and compare their own models. Our dataset and evaluation code are publicly available: https://bit.ly/frmt-task