M2G-Eval: Enhancing and Evaluating Multi-granularity Multilingual Code Generation
This provides a more comprehensive evaluation benchmark for code LLMs, addressing limitations in existing single-granularity, limited-language assessments, though it is incremental in enhancing evaluation methodology rather than proposing new model architectures.
The authors tackled the problem of evaluating code generation in large language models (LLMs) by introducing M2G-Eval, a multi-granularity, multilingual framework spanning 18 programming languages with 17K+ training tasks and 1,286 test instances, which revealed a difficulty hierarchy from Line-level (easiest) to Class-level (most challenging) and identified performance gaps and cross-language correlations.
The rapid advancement of code large language models (LLMs) has sparked significant research interest in systematically evaluating their code generation capabilities, yet existing benchmarks predominantly assess models at a single structural granularity and focus on limited programming languages, obscuring fine-grained capability variations across different code scopes and multilingual scenarios. We introduce M2G-Eval, a multi-granularity, multilingual framework for evaluating code generation in large language models (LLMs) across four levels: Class, Function, Block, and Line. Spanning 18 programming languages, M2G-Eval includes 17K+ training tasks and 1,286 human-annotated, contamination-controlled test instances. We develop M2G-Eval-Coder models by training Qwen3-8B with supervised fine-tuning and Group Relative Policy Optimization. Evaluating 30 models (28 state-of-the-art LLMs plus our two M2G-Eval-Coder variants) reveals three main findings: (1) an apparent difficulty hierarchy, with Line-level tasks easiest and Class-level most challenging; (2) widening performance gaps between full- and partial-granularity languages as task complexity increases; and (3) strong cross-language correlations, suggesting that models learn transferable programming concepts. M2G-Eval enables fine-grained diagnosis of code generation capabilities and highlights persistent challenges in synthesizing complex, long-form code.