On Evaluating Multilingual Compositional Generalization with Translated Datasets
This addresses the need for robust benchmarks in cross-lingual compositional generalization for NLP researchers, though it is incremental as it builds on existing datasets and methods.
The paper tackled the problem of evaluating multilingual compositional generalization by showing that neural machine translation introduces semantic distortions in existing benchmarks, and they created a rule-based translation of the MCWQ dataset to Chinese and Japanese, revealing that models still struggle with cross-lingual generalization despite this improvement.
Compositional generalization allows efficient learning and human-like inductive biases. Since most research investigating compositional generalization in NLP is done on English, important questions remain underexplored. Do the necessary compositional generalization abilities differ across languages? Can models compositionally generalize cross-lingually? As a first step to answering these questions, recent work used neural machine translation to translate datasets for evaluating compositional generalization in semantic parsing. However, we show that this entails critical semantic distortion. To address this limitation, we craft a faithful rule-based translation of the MCWQ dataset from English to Chinese and Japanese. Even with the resulting robust benchmark, which we call MCWQ-R, we show that the distribution of compositions still suffers due to linguistic divergences, and that multilingual models still struggle with cross-lingual compositional generalization. Our dataset and methodology will be useful resources for the study of cross-lingual compositional generalization in other tasks.