MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages
This addresses the barrier for non-English-speaking developers in code generation, though it is incremental as it extends an existing dataset to new languages.
The authors tackled the English-centric bias in code generation by creating MCoNaLa, a multilingual dataset with 896 NL-code pairs in Spanish, Japanese, and Russian, and found that state-of-the-art systems perform significantly worse on these languages compared to English.
While there has been a recent burgeoning of applications at the intersection of natural and programming languages, such as code generation and code summarization, these applications are usually English-centric. This creates a barrier for program developers who are not proficient in English. To mitigate this gap in technology development across languages, we propose a multilingual dataset, MCoNaLa, to benchmark code generation from natural language commands extending beyond English. Modeled off of the methodology from the English Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896 NL-code pairs in three languages: Spanish, Japanese, and Russian. We present a quantitative evaluation of performance on the MCoNaLa dataset by testing with state-of-the-art code generation systems. While the difficulties vary across these three languages, all systems lag significantly behind their English counterparts, revealing the challenges in adapting code generation to new languages.