Beyond BLEU: A Semantic Evaluation Method for Code Translation
For researchers and practitioners evaluating code translation models, this work provides a more accurate semantic metric, revealing the inadequacy of BLEU for this task.
The paper proposes a semantic evaluation method for code translation that measures functional correctness rather than syntactic similarity, showing that LLM-based decompilers significantly outperform heuristic ones while BLEU scores have negligible correlation with semantic correctness (r = -0.127 to 0.354).
Code translation is one of the core capabilities of LLMs. However, evaluating the correctness of translations remains difficult, as commonly used metrics such as BLEU measure only syntactic similarity, disregarding program semantics. We propose a novel evaluation methodology for code translation tasks, emphasizing semantic equivalence over surface-level string similarity. Our approach applies established compiler testing methodology to a new domain, allowing the assessment of an LLM fine-tuned for binary lifting tasks (i.e. decompiling binaries to higher-level representations). We introduce a semantic correctness score, defined as the proportion of translations that produce correct execution outcomes, and demonstrate its application by evaluating LLM-based and heuristic decompilers. Our findings show that LLM-based approaches significantly outperform heuristic ones, while BLEU scores show negligible correlation with semantic correctness (r = -0.127 to 0.354), demonstrating that syntactic metrics fail to predict functional accuracy.