MULTITEXTEDIT: Benchmarking Cross-Lingual Degradation in Text-in-Image Editing
For researchers and developers of text-in-image editing systems, this benchmark reveals systematic cross-lingual failures that existing English-centric benchmarks miss, highlighting the need for language-aware evaluation.
MULTITEXTEDIT introduces a 3,600-instance benchmark across 12 languages to measure cross-lingual degradation in text-in-image editing, finding that all 12 tested models show pronounced degradation, especially for Hebrew and Arabic, with errors concentrated in text accuracy and script fidelity rather than coarse structure.
Text-in-image editing has become a key capability for visual content creation, yet existing benchmarks remain overwhelmingly English-centric and often conflate visual plausibility with semantic correctness. We introduce MULTITEXTEDIT, a controlled benchmark of 3,600 instances spanning 12 typologically diverse languages, 5 visual domains, and 7 editing operations. Language variants of each instance share a common visual base and are paired with a human-edited reference and region masks, isolating the language variable for cross-lingual comparison. To capture script-level errors that coarse text-matching metrics miss, such as missing diacritics, reversed RTL order, and mixed-script renderings, we introduce a language fidelity (LSF) metric scored by a two-stage LVM protocol that first traces the edited target text and then judges it in isolation, reaching a quadratic-weighted \k{appa} of 0.76 against native-speaker annotators. Evaluating 12 open-source and proprietary systems with LSF alongside standard semantic and mask-aware pixel metrics, we find pronounced cross-lingual degradation for every model, largest on Hebrew and Arabic and smallest on Dutch and Spanish, and concentrated in text accuracy and script fidelity rather than in coarse structural dimensions. We also uncover a pervasive semantic and pixel mismatch, where outputs preserve global layout and background fidelity yet distort script-specific forms.